Further Improvement

Click in the menu to navigate to the desired section

Foreword I Opening: Monique Vogelzang Further Improvement: A Closer Look II Interview: VSNU, VH, NRTO III Message from the minister The conference, background and good practices IV Interview: ISO V Workshops VI Interview: Hobéon, QANU, AeQui VII Infomarket VIII Presentation: Jan Anthonie Bruijn IX Panel discussion X Closing address: Monique Vogelzang 10+ Literature and links

Further Improvement a coser look 1 Foreword

The value of a diploma must be beyond to reading the reports and interviews, we all doubt. Both students and society have recommend that you follow the links under Foreword a right to be assured of that. This is an ‘more information’. These form a repository issue in which examination boards play of useful documents, good practices and I Opening: Monique Vogelzang a crucial role. On 20 May 2015 the Inspec- tools by other examination boards and II Interview: VSNU, VH, NRTO torate of Education held the Further organisations in higher education. Improvement conference, which shares III Message from the minister Monique Vogelzang its title with the inquiry by the Inspec- This e-magazine is the fruit of collaboration Inspector-General of Education IV Interview: ISO torate into the performance of higher-ed- between many workshop facilitators, all of ucation examination boards. The results whom seek to contribute to the quality of V Workshops were published in the spring of this year. higher education from their own position VI Interview: Hobéon, QANU, AeQui The report provides an outline of where of responsibility. I offer them my thanks, VII Infomarket examination boards currently stand, and and hope that after the conference, this HOW TO READ THIS E-MAGAZINE indicates where further improvements can e-magazine will also contribute to further VIII Presentation: Jan Anthonie Bruijn be made. improvement among examination boards. Browse through the text, or click the menu on the left to IX Panel discussion proceed to the next section. To view a workshop, click Over 600 participants from research univer- Monique Vogelzang ‘Workshops’ in the left-hand menu and then select the X Closing address: Monique Vogelzang sities, universities of applied sciences and Inspector-General of Education relevant workshop from the list. You can also find the Literature and links non-government-funded institutions workshops by simply browsing attended the conference, which owed June 2015 through the magazine. its success to the exchange of a wide range of experiences, active participation The top-right corner of every workshop (except the by attendees in one or more of the 23 first, which is of a general nature) outlines the relevance workshops, and presentations by countless of the workshop to the Further Improvement report. interesting speakers. Links to relevant documents can be accessed at the This e-magazine aims to serve both as a bottom left of each section or workshop, under ‘More 10+ report of the conference and as a reference information’. Click the documents to view them. The work for all examination boards and back of the magazine contains an overview of the key

managers in higher education. In addition EXAMINATION BOARDS background documents, including links. in higher education Opening a link will close this magazine. If you want to be able to switch between the magazine and one of the background documents, download the PDF first and the link will open in your browser automatically. We recommend downloading a new PDF each time, as small improvements will be made from time to time.

If you would prefer to read the paper version of this e-magazine, simply print it as you would with any other document. You will not be able to use the links, however.

Further Improvement a coser look 2 ‘Welcome to the conference on the I am pleased to introduce you to Martine Inspectorate’s report concerning the Pol, the project manager of this inquiry. If performance of examination boards. The you have any questions about the inquiry, amount of interest in this conference was or if you wish to organise a presentation enormous. We were expecting around 300 by the Inspectorate as a follow-up to people, but soon exceeded that number. this conference at your own institution, After only ten days we had received over please contact Martine via 500 registrations, and we ultimately cut [email protected].’ off the applications at 650. Otherwise we would have had to hire Ahoy... Around Erik Martijnse Foreword 80 of our attendees come from non-gov- Day’s chairman ernment-funded higher education I Opening: Monique Vogelzang institutions, 330 from funded universities II Interview: VSNU, VH, NRTO of applied sciences and 160 from research universities. And, last but not least, the 65 III Message from the minister other participants: our workshop facili- IV Interview: ISO tators and representatives from other V Workshops organisations. VI Interview: Hobéon, QANU, AeQui VII Infomarket VIII Presentation: Jan Anthonie Bruijn ‘Welcome to the IX Panel discussion X Closing address: Monique Vogelzang conference on the Literature and links Inspectorate’s 10+ report’ I would like to step back in time for a moment, to 27 March 2012 and the motion submitted by Hans Beertema. A majority in the Dutch Lower House of Parliament had lost all confidence in the value of higher-education diplomas. The proposition was to banish examinations from institutions, and replace them with final examinations at national level. Just like driving exams, a centralised body would be responsible for higher-education diplomas. Bye-bye examination boards: that’s how low confidence had dropped. And where are we today?

Further Improvement a coser look 3 Foreword I Opening: Monique Vogelzang II Interview: VSNU, VH, NRTO III Message from the minister IV Interview: ISO V Workshops VI Interview: Hobéon, QANU, AeQui VII Infomarket VIII Presentation: Jan Anthonie Bruijn IX Panel discussion X Closing address: Monique Vogelzang 10+ Literature and links

Further Improvement a coser look 4 Opening: Towards further improvement Monique Vogelzang

Monique Vogelzang that their performance has improved over the last six years, especially in higher Foreword The Further Improvement* report professional education. Testing is higher describes the current situation with up on the agenda, and examination I Opening: Monique Vogelzang respect to examination boards. In her boards are formulating guidelines for the II Interview: VSNU, VH, NRTO opening address, Inspector-General administration and evaluation of tests. They have also become more independent: III Message from the minister Monique Vogelzang outlined the main points.. their members include fewer and fewer IV Interview: ISO managers, and are almost always appointed V Workshops Do you all remember your first swimming by the executive board. Lastly, expertise has diploma? First the cold and nervous wait increased: many examination boards have VI Interview: Hobéon, QANU, AeQui at the edge of the pool, then a minute of taken targeted steps to further professional VII Infomarket treading water, and then finally swimming development. to the other end with your clothes on, heavy VIII Presentation: Jan Anthonie Bruijn as lead. But if you made it, the big moment Developmental stage IX Panel discussion arrived: your first diploma! It was the proof Although major steps have been taken in recent years, we can see that examinations X Closing address: Monique Vogelzang that you could do it, the reward. boards are still in the developmental stage. Literature and links There are two major areas for improvement. So I know that we all have at least one Firstly, there needs to be a greater focus on diploma, but many more probably followed testing. The guidelines for creating tests can after that. And where do you actually keep be refined, and monitoring of compliance I them all? Has anybody ever asked you for can also be improved – particularly with the physical proof of your graduation? guidelines pertaining to fraud. Some Employers rarely do. And yet the existence examination boards fail to properly monitor of diplomas is important to employers, the guidelines’ effectiveness. A second area employees and to society: they are the for improvement is the appointment of 10+ proof that as a graduate, you possess a examiners. Although examiners usually are certain level of knowledge and skills. The appointed, this is often done implicitly: value of diplomas must therefore remain one third of examiners actually claim not to uncontested. But how do we ensure their know whether the examination board has value? In higher education institutions it is appointed them or not. Who is responsible the task and responsibility of examination if examiners fail to perform up to standard? boards to ensure that the examination The examination board, or their manager? process runs as it should, and that the diploma represents an appropriate ‘reward’ This area therefore requires further MORE INFORMATION for the achievement of graduation. improvement. A prerequisite for improvement is administrative support. Further Improvement* As the Inspectorate, we investigated how Now that the flexibility of higher education examination boards have developed is increasing, the quality assurance role of Bookkeeper or watchful eye? since the previous inquiry in 2008 titled examination boards is becoming more and ‘Bookkeeper or watchful eye?’ (Boekhouder more important. Examination boards are of wakend oog?). Our studies have shown working hard to fulfil >>

Further Improvement *English translation a coser look 5 Opening Monique Vogelzang

>> this role effectively, however quality assurance is a task that involves the entire institution. Executive boards must give ‘There needs to be a greater Foreword examination boards enough time for their I Opening: Monique Vogelzang professional development and for the II Interview: VSNU, VH, NRTO performance of their duties. focus on testing. III Message from the minister Know thyself IV Interview: ISO Society needs proper examination boards. To help improve their position, I would The guidelines for creating V Workshops like to give the examination boards some VI Interview: Hobéon, QANU, AeQui unsolicited advice: get to know yourselves, VII Infomarket and use the tools you have at your disposal. tests can be refined, and Every board has its own concerns and VIII Presentation: Jan Anthonie Bruijn issues. Investigate whether you are fulfilling IX Panel discussion all the required duties, and whether you do so effectively and productively. Use our monitoring of compliance X Closing address: Monique Vogelzang inquiry and other sources as a performance Literature and links checklist. Examination boards can also learn from each other – developments by can also be improved’ one board may well be of use to another. Exchange knowledge, work together, and share good practices. Umbrella organi- I sations will also play a key role in this respect.

We jumped in the deep end a long time ago, and the swimming diplomas are in. Now it is time to learn to freestyle.

EXAMINATION BOARDS in higher education MORE INFORMATION Further Improvement* Bookkeeper or watchful eye?

Further Improvement a coser look 6 *English translation Interview ‘Students should be VSNU, VH, NRTO proud of their diplomas’

Interview with Karl Dittrich (Association of Universities in the Foreword Netherlands, VSNU), Thom de I Opening: Monique Vogelzang Graaf (Netherlands Association II Interview: VSNU, VH, NRTO of Universities of Applied III Message from the minister Sciences, VH) and Hans Hillen Karl Dittrich President, Association of Universities in the IV Interview: ISO (Dutch Council for Training and Netherlands V Workshops Education, NRTO) VI Interview: Hobéon, QANU, AeQui VII Infomarket The Further Improvement* report has full steam ahead, by training examination shown that examination boards have board members and developing informative VIII Presentation: Jan Anthonie Bruijn made considerable progress in recent materials.’ By way of illustration, he handed IX Panel discussion years. What have they achieved? And, Monique Vogelzang the first copy of the second edition of the Examination Committee X Closing address: Monique Vogelzang more importantly: where will the focus lie in the years ahead? Inspector-General Guide (Handreiking examencommissies), Literature and links Monique Vogelzang interviewed umbrella hot off the presses from that very morning. organisation presidents Karl Dittrich (VSNU), Thom de Graaf (VH) and Hans Freedom Hillen (NRTO) about the results of the In answer to the question of where inquiry. the focus of universities of applied II sciences should lie in the years ahead, The three gentlemen on stage all agreed: De Graaf named network formation and Thom de Graaf the improved internal orientation of President, Netherlands Association of although there has been major progress, Universities of Applied Sciences the continued development of examination examination boards within the universities, 10+ boards remains a priority. Thom de Graaf: with specific attention to professional ‘Universities of applied sciences are moving development. Hans Hillen stressed the in the right direction, but that’s no reason independence of examination boards.>> for us to sit back and relax. We’re still going ‘The report allows for MORE INFORMATION 2015 Guide sufficient leeway, which the NRTO Service Document examination boards need to Further Improvement* Hans Hillen President, Dutch Council for Training take advantage of’ and Education Further Improvement *English translation a coser look 7 >> ‘Independence is necessary for an Interview VSNU, objective assessment of lecturers’ and students’ performance.’ Professionals VH, NRTO also require freedom, he said, in order to structure their study programmes. ‘Although we are seeing more and more standardisation in this country, creativity in education is just as important. The Further Improvement report allows for sufficient leeway, which the examination boards need Foreword to take advantage of.’ I Opening: Monique Vogelzang Karl Dittrich took a similar view. ‘I read II Interview: VSNU, VH, NRTO the report with a certain degree of surprise III Message from the minister and admiration. We are almost on our way to creating a perfect society! But at IV Interview: ISO the same time, we need to make sure that V Workshops the great does not become the enemy of the good. Shouldn’t we be moving away VI Interview: Hobéon, QANU, AeQui from protocols, and towards increased VII Infomarket faith in the profession?’ Thom de Graaf did not believe that the report places VIII Presentation: Jan Anthonie Bruijn any undue emphasis on regulations. ‘I IX Panel discussion think that people in research-oriented X Closing address: Monique Vogelzang education, higher professional education and non-government-funded education are Literature and links all colleagues who can learn a lot from each other. The report includes suggestions for doing so.’

Confidence restored II We also need to consider everything that is moving in the right direction, said Karl Dittrich. ‘We should be pleased with what we are doing right, and find out what we 10+ can do better. Society’s confidence in education has been restored, and we need to let people know we are happy about it.’ The others concurred. Hillen: ‘Students should be proud of their diplomas.’

MORE INFORMATION 2015 Guide NRTO Service Document Further Improvement*

Further Improvement a coser look 8 *English translation Interview VSNU, VH, NRTO

Thom de Graaf offers Monique Vogelzang the first copy of the Guide Further Improvement a coser look 9 Message from ‘Things are going well, but there is room the minister for further improvement’

Foreword I Opening: Monique Vogelzang II Interview: VSNU, VH, NRTO III Message from the minister IV Interview: ISO V Workshops VI Interview: Hobéon, QANU, AeQui VII Infomarket VIII Presentation: Jan Anthonie Bruijn Jet Bussemaker IX Panel discussion Minister of Education, Culture and Science X Closing address: Monique Vogelzang Literature and links Much to her regret, Minister Jet Busse- maker was unable to attend the conference today. She did send a video message to highlight the importance of examina- III tion boards. The minister was pleased with the improved levels of quality assurance for all types of interim and final examinations. Examination boards are increasingly able to guarantee the value of the diplomas issued. However, further steps can – and must – be taken. Some examination boards and programme managers, for example, have no clear view of their powers or of the statutory frameworks in which they operate. To perform effectively, examination boards must also be given adequate support for the fulfilment MORE INFORMATION of their duties and for professional development. In many cases, that support is lack- Letter to the Lower House ing. Lastly, the minister provided some food for thought: what areas for improvement can you identify within your own board? Where do the priorities lie in the professional development of examination boards and examiners? And how will you carry on tomorrow with what you have discussed today? Further Improvement a coser look 10 Interview: Students demand ISO improved testing

Interview with Yvonne Rouwhorst mistakes. This aspect receives insufficient and Lisanne van Kessel (ISO) focus in the Inspectorate’s report.’ Foreword What is your idea of proper testing? I Opening: Monique Vogelzang Although students would rather pass ‘An effective examination needs to satisfy II Interview: VSNU, VH, NRTO an exam than fail, they greatly value many requirements, but the most important objective testing. During the conference is that it is part of a well-considered testing III Message from the minister we spoke to Yvonne Rouwhorst and policy in line with the study programme’s IV Interview: ISO Lisanne van Kessel from the Dutch core values. So the structure, content and National Student Association (ISO) V Workshops timing of the exam need to fit within the about testing and examination boards. programme as a whole, and be relevant to VI Interview: Hobéon, QANU, AeQui Rouwhorst is an ISO board member the skills students will need when they enter whose remit includes the legal status of VII Infomarket “the real world” with their degree in hand.’ students, and the quality of teaching. VIII Presentation: Jan Anthonie Bruijn As a project officer, Van Kessel conducts IX Panel discussion research into topics such as testing and What do you want from an examination feedback. board? X Closing address: Monique Vogelzang ‘The job of an examination board is to Literature and links But why is testing so important to ensure that tests and testing policy match students? the programme’s core values. In the future, ‘Students place great importance on proper the ISO plans to move towards learning testing. Any diploma they might earn is only outcomes, tests and assessments that are worth something if it represents a certain formulated independently of any particular IV level of quality. And because a diploma learning track. This will give students is the sum total of all the programme greater freedom to design their course components, each of which often concludes of study in a manner that suits them. with a test or examination, it is important Examination boards play a major role in for the testing to be of a proper quality. this respect: they evaluate whether students Formative testing – the ongoing process ultimately satisfy the exit qualifications of collecting information on learning set out in the Teaching and Examination results – is also of great importance, and Regulations.’ concentrates on providing students with >> information on their current progress and about how they can learn from their

MORE INFORMATION ISO bottleneck report ‘Any diploma is only worth something if it represents a certain level of quality’

Further Improvement 11 >> The Further Improvement report states of receipt. Deadlines are missed, and where process that teaching staff can follow online Interview: that communication with students and and how the complaint should be answered and supervise if necessary. Feedback is a the transparency of examination boards’ is unclear. There is definitely plenty of room continuous process, not a snapshot.’ processes are points for improvement. for improvement.’ ISO Would you agree? ‘The ISO’s bottleneck report (knelpun- Lastly, we know that you are working on tennotitie) that has now prompted a a report dealing with feedback. Can you legal-status inquiry commissioned by reveal any details? Minister Bussemaker, raised the issue of ‘The ISO is currently investigating best ‘Digital education procedures and information provision. practices for testing and feedback. In Foreword Careless procedures regarding examination the future, feedback will become a more boards and inadequate information for important and valuable component of I Opening: Monique Vogelzang will allow teaching students concerning their status in the the study process. Developments such II Interview: VSNU, VH, NRTO process are fairly commonplace. For as digital education and tests will allow III Message from the minister example, examination boards often contain teaching staff to provide feedback much the very lecturers who are the subject of faster and more directly. The ISO is arguing staff to provide IV Interview: ISO complaints of a failure to act objectively. for teaching materials that are more open V Workshops Students’ study progress can also be delayed and include integrated feedback: they yield due to slow-moving appeals procedures. better results. Feedback should be more feedback much VI Interview: Hobéon, QANU, AeQui Sometimes the procedure for a thesis than just a grade. Research has also shown VII Infomarket re-assessment by a second examiner, for that students’ need for feedback decreases example, is convoluted or simply unknown. once they have received a grade, and peer VIII Presentation: Jan Anthonie Bruijn The ISO is very concerned that procedures feedback will take on a more important faster’ IX Panel discussion often lack transparency – students do not role. Students from all kinds of institutions X Closing address: Monique Vogelzang know where to turn, and if they do submit – including internationally – will be able to a complaint, they receive no confirmation provide each other with feedback online, a Literature and links

MORE INFORMATION ISO bottleneck report

Further Improvement a coser look 12 Workshops Overview of workshops The conference included a total of 23 workshops. Click on the workshop number to skip to the relevant page.

1. Further Improvement Teaching Café 12. Calibration of final assignment submissions in higher education Martine Pol, Anne-Marie van Rijsbergen, Annelies Bon, Jan Willem Roodenberg Marlies van Beek (Cito) Foreword (Inspectorate of Education) I Opening: Monique Vogelzang 13. The interpretation of the Higher Education and Research Act II Interview: VSNU, VH, NRTO 2. The Examination Board Guide: an authoritative document (WHW) in relation to examination boards Roeland Smits, Caroline Stam, Jokelyne Gerritsen (VH) Patrick Leushuis, Trees Ruijgrok (Ministry of Education, Culture and Science) III Message from the minister IV Interview: ISO 14. The examination board in the legal precedents of the Higher V Workshops 3. Quality assurance and the performance of examination boards Education Appeals Tribunal Martin Kropff (Wageningen University) Willem Beijk, Jan Nijenhof, Ben Olivier (CvBHO) VI Interview: Hobéon, QANU, AeQui VII Infomarket 4. Quality quantified: the building-blocks of a (centralised) 15&16. Valid and reliable testing in art education VIII Presentation: Jan Anthonie Bruijn examination board Anton Neggers (Fontys), Jackelien ter Burg (ArtEZ), Jan Wirken Martine Pieters, Justus Tengbergen, Nico Scheeres, Markus Verbeek (NCOI) (Zuyd University of Applied Sciences) IX Panel discussion X Closing address: Monique Vogelzang 5. Examination boards: a question of culture? 17. Testing: not less, just different Literature and links Sibe Doosje, Marie-Jet Fennema (Utrecht University) Lex Jansen (handicap + studie Expertise Centre)

6. Collaboration on test quality: rules of engagement 18. Fraud: from practice to policy Mieke Jaspers, Els van Zijl (Fontys Universities of Applied Sciences) Ludo van Meeuwen (Eindhoven University of Technology), Arie de Wild V (Rotterdam University of Applied Sciences)

7. Quality assurance: from regulations to understanding 19. The examination board, the management and administration: Edith Hooge (TIAS, Tilburg University) a ‘dynamic’ relationship Jan Wiss (Inholland)

8. The examination board through the eyes of a student 20. Dilemmas in the work of examination board members Klaasjan Boon (LSVb) Frans de Vijlder (HAN University of Applied Sciences)

9. The examination board as a professional community 21.Research universities and universities of applied sciences: John Huizinga (TIAS, Utrecht University of Applied Sciences), Frans de Vijlder learn from each other! (HAN University of Applied Sciences) Susan Voogd (VU)

10. Tests, examination and the role of examination boards in higher 22. Testing and assessment, and the limited evaluation education accreditation of study programmes / Sietze Looijenga (QANU), Hester Minnema (Leiden Univer- Wienke Blomen, Frank Hendriks, Robert Stapert (Hobéon) sity), Ivo van Stokkum (VU University Amsterdam)

11. How can institutions work on the test expertise of their examiners? 23. Ensuring quality and standards in joint higher-education pro- Remko van der Lei, Brenda Aalders (Hanze University of Applied Sciences, Groningen) grammes with international partner institutions Peter Wieringa, Marinke Sussenbach (TU Delft) Further Improvement a coser look 13 Workshops

Further Improvement a coser look 14 Workshops Further Improvement Teaching Café

Martine Pol, Anne-Marie van an examination board member, for fraud Rijsbergen, Annelies Bon, Jan to go unnoticed in such cases? Discussion Foreword also turned to the pilot conducted by the ‘Effective Willem Roodenberg (Inspectorate Inspectorate at Fontys as part of the inquiry, I Opening: Monique Vogelzang of Education) in which the two institutions trialled a II Interview: VSNU, VH, NRTO procedure that makes supervision both supervision more effective and less labour-intensive. III Message from the minister One important way to achieve further improvement in examination boards IV Interview: ISO is through the mutual exchange of Does examiners’ test expertise require further improvement? means that the V Workshops experiences and entering into dialogues with one another. What exactly are At the next table, there was some VI Interview: Hobéon, QANU, AeQui the core tasks of examination boards? controversy surrounding examiners’ testing VII Infomarket How does the Inspectorate conduct its skills and the support they receive to Inspectorate supervision? What expectations can develop them. Many examination boards VIII Presentation: Jan Anthonie Bruijn be placed on examiners? Members of called to make this aspect a higher priority IX Panel discussion examination boards discussed these and during the inquiry, while many examiners claimed to be generally quite satisfied should be open to X Closing address: Monique Vogelzang other topics at the Teaching Café with their colleagues and with the inspectors with the amount of time, support and Literature and links who carried out the Further Improvement the training opportunities they received inquiry. We present some of their for examination purposes. So why do such invitations’ discussions below. the opinions of examination boards and examiners differ so greatly? How can this V.1 At the café’s four tables a lively debate discrepancy be explained? Special publication was conducted. At one table, higher The results of the questionnaire have education inspectors Martine Pol This difference in perspective came up a left examination board members a little (Further Improvement project manager), lot during the discussion: examination bewildered, because they do hear lecturers Anne-Marie van Rijsbergen, Annelies Bon boards have a different focus and interest, complaining that they have too little 10+ and Jan Willem Roodenberg were busy and a better overall view of the programme time for testing. Some lecturers actually answering questions on the inquiry. Topics than the examiners, allowing them to see report that they recycle old exams for this of discussion included the confidentiality the discrepancies more easily. Another very reason. The results of the inquiry of the questionnaire, the selection of possible contributing factor is the fact that would therefore seem to require more institutions and the status of the report’s not all examiners view teaching as their in-depth examination in this respect, and ‘good practices’ (see the ‘Inspectorate’s most important task – lecturers at research the Inspectorate is considering releasing procedures’ link for the answers to these universities in particular tend to focus on a special publication on the subject of questions). research, making testing somewhat of a examiners, incorporating the input from forgotten step-child. Some lecturers are the discussions at this table. MORE INFORMATION Participants in the discussions also had the also less inclined to spend time on testing opportunity to respond to the report itself. because they do not enjoy it, or prefer to In the future, institutions may organise The Inspectorate’s procedures One of the tables, for example, discussed dedicate more time to teaching and student their own events on ways to improve supervision. examination boards. The inspectors have Fontys pilot the question of whether the limited rates of fraud among such large student expressed their willingness to contribute, Good practices numbers are really as noteworthy as the emphasising that effective supervision report claims. Surely it is logical, argued means being open to such invitations. Further improvement* Further Improvement *English translation a coser look 15 Workshops The Examination Board Guide: an authoritative document

Roeland Smits and Caroline Stam and Roeland Smits and Caroline Stam demon- RELEVANCE TO ‘FURTHER IMPROVEMENT’ (Netherlands Association of Uni- strated how it can be used to aid the implemen- In its report, the Inspectorate advises examination Foreword tation of the Further Improvement* recom- boards to develop a clear, joint interpretation of versities of Applied Sciences), and mendations. the applicable legislation. The Examination Board I Opening: Monique Vogelzang Jokelyne Gerritsen (Windesheim Guide by the Netherlands Association of Univer- II Interview: VSNU, VH, NRTO University of Applied Sciences) The Examination Board Guide has developed into sities of Applied Sciences (VH) can be a valuable an authoritative document, and even earned a III Message from the minister asset in this regard. This workshop looked at reference in the Council of State’s recommen- relevant points from the Guide that can be used to The Further Improvement* report has shown IV Interview: ISO dations concerning the Improved Governance implement the recommendations from the Inspec- that funded universities of applied sciences V Workshops (Higher Education) Act. This workshop devoted torate’s report. are performing well in terms of implementing particular attention to the Inspectorate’s recom- VI Interview: Hobéon, QANU, AeQui the Improved Governance (Higher Education) mendation to produce an unambiguous interpre- Act. Since the previous inquiry (Bookkeeper or VII Infomarket tation of the statutory framework. The Guide can watchful eye?) into the role and performance contribute to this desired process of clarification, VIII Presentation: Jan Anthonie Bruijn of examination boards, higher professional making it relevant to the focus areas outlined in education has made considerable progress. IX Panel discussion Further Improvement*. For example, it discusses The Examination Board Guide published by the question of whether examination boards are X Closing address: Monique Vogelzang the Netherlands Association of Universities authorised to revoke an examiner’s appointment Literature and links of Applied Sciences (VH, 2011) made a major – a question that various examination boards contribution to this process. On 20 May 2015, grapple with, the report showed. (Incidentally, a second edition of the Guide was released, the Guide’s answer to this question is ‘yes’.) 10+V.2

MORE INFORMATION 2015 Guide External Validation Report Assessment is not a formula Responsible testing Further improvement* Further Improvement Bookkeeper or watchful eye? a coser look 16 *English translation Workshops ‘We like to get people thinking’ Interview with Roeland Smits & Caroline Stam (Netherlands Association of Universities of Applied Sciences)

How does the second edition of the we will have trained a total of over 400 Foreword Examination Board Guide differ from examination board members, all of whom I Opening: Monique Vogelzang the first? also share their knowledge and experience ‘The second edition of the Guide is both with their colleagues. Each session consists II Interview: VSNU, VH, NRTO the same as, and totally different from, of three sections: testing, legal issues and the first. The structure, template, target the position of the examination board III Message from the minister Caroline Stam audience and approach are all the same. within the organisation. We incorporated IV Interview: ISO Legal policy adviser, Netherlands The difference lies in the fact that we trainers’ experiences gained during the Association of Universities of Applied V Workshops have included a number of legislative sessions into the Guide as much as we Sciences & HU University of Applied Sciences Utrecht VI Interview: Hobéon, QANU, AeQui changes that have been adopted since the could. Just like the first edition of the first edition, as well as the results of the Guide, however, the second edition will VII Infomarket Further Improvement* report. There is too continue to focus on the statutory tasks and VIII Presentation: Jan Anthonie Bruijn little time to list all the differences here: authorisations of examination boards.’ suffice it to say that the Guide has been IX Panel discussion updated in a range of areas, including the What is your opinion of the conference, X Closing address: Monique Vogelzang relationships between examination boards and what is your contribution? and institutional (and other) management, ‘Events like this are not only fun, they’re Literature and links the boards’ role in instances of fraud, and very instructive too. Our interactive guaranteeing the exit levels for diplomas.’ workshops are educational to us as well. We learn from the participants. The most Did you incorporate professional important thing, we believe, is to get V.2 development experiences when people thinking. When they get home preparing the second edition? tonight, it is my hope that people will go ‘Certainly. The Association has already through the workshop again in their minds run a total of seven training sessions for and critically ask themselves how they examination board members. We definitely intend to follow up on it. We hope to send Roeland Smits 10+ expect an eighth, and possibly even a the participants home full of energy, just Senior legal affairs and quality policy ninth or tenth group to follow. With 45 like we are.’ adviser, Netherlands Association of participants in each group, this means that Universities of Applied Sciences ‘We have included the MORE INFORMATION 2015 Guide results of the inspection External Validation Report Assessment is not a formula report in our update of Responsible testing Further improvement* the Guide’ Further Improvement Bookkeeper or watchful eye? a coser look 17 Workshops Quality assurance and the performance of examination boards

Martin Kropff underway on standardising the process RELEVANCE TO ‘FURTHER IMPROVEMENT’ (Wageningen University) for converting foreign grades into those In its report, the Inspectorate advises all parties to Foreword used at Wageningen University, as research jointly decide on a clear and transparent position has shown that international interim and for examination boards within their institutions. I Opening: Monique Vogelzang Establishing, documenting and final examinations are graded more highly It is also important for the executive board to use II Interview: VSNU, VH, NRTO implementing testing policy is a complex than in the Netherlands. For this reason, the examination board’s annual report to enhance process, especially at a university with an assessor from the home university both the board’s position and the quality of exam- III Message from the minister such a strong international perspective largely determines the grade to be awarded inations. During this workshop and follow-up in- IV Interview: ISO as Wageningen. Still, the programmes for examinations or internship reports terview, Wageningen University Rector Magnificus at Wageningen University have received V Workshops completed abroad. International assessors Martin Kropff explained his university’s approach positive evaluations for years, the more and supervisors play an advisory role, but to these aspects. VI Interview: Hobéon, QANU, AeQui ‘difficult’ programmes in particular. Why have no decisive power in this process. is this, and what role do the university’s VII Infomarket Standardisation of this practice has ensured four examination boards play in the a uniform assessment procedure. VIII Presentation: Jan Anthonie Bruijn process? Rector Magnificus Martin Kropff IX Panel discussion presented his views. Some attendees struggled with the problem of how to properly and remotely assess X Closing address: Monique Vogelzang Wageningen University offers 19 BSc and students who are completing internships Literature and links 28 MSc programmes to more than 10,000 abroad. Visits to the host organisation are students from over 100 countries. The not unheard of, but come at great expense. high level of programme diversity and The audience advised increased use of video complexity requires not one, but four technology. A second suggestion was the examination boards, each with qualified development of joint tests, which provide a V.3 members. Strikingly enough, the university clearer picture of internship quality. >> never has any major problems finding new examination board members. The university’s good reputation, and the fact that 10+ the examination boards receive adequate support, help to ensure sufficient interest. The high number of positively evaluated ‘The audience advised MSc programmes (21) is a testament to the examination boards’ effective performance. using video technology International testing hardly problematic Many questions were asked by the audience during the well-attended presentation, MORE INFORMATION with particular attention to Wageningen in the assessment of University’s international focus on quality Rules and Regulations assurance. How do they guarantee the quality of interim and final examinations students completing Testing and Assessment Policy completed abroad? Examination board members at Wageningen maintain regular Workshop Presentation contact on this issue with teaching staff in other countries, Kropff said. Work is also internships abroad’

Further Improvement a coser look 18 >> Plenty of room for improvement Workshops After his presentation, Kropff put forward several propositions for the audience to debate. But the nature and importance of the subject at hand was only highlighted by the fact that the propositions could not be dealt with: the audience continued to ask questions, which Kropff answered enthusiasti- Voorwoord cally. All those present acknowledged the I Opening Monique Vogelzang importance of good-quality education and proper testing. And although things were II Interview VSNU, VH, NRTO moving in the right direction, there was still III Boodschap van de minister plenty of room for improvement. Thankful- ly everyone was prepared to pitch in. IV Interview ISO V Workshops VI Interview Hobéon, QANU, AeQui VII Informatiemarkt VIII Presentatie Jan Anthonie Bruijn IX Paneldiscussie X Slotwoord Monique Vogelzang Literatuur en links 10+V.3

MORE INFORMATION Rules and Regulations Testing and Assessment Policy Workshop Presentation

Further Improvement a coser look 19 Workshops ‘The executive board and examination boards must work in tandem’ Interview with Martin Kropff (Wageningen University)

What changes have affected the examinations is a vast and complex task, examination boards at Wageningen and substantial growth in student numbers Foreword University in recent years? is increasing our workloads even more. I ‘In addition to student-related activities, resolved to make sure that examination I Opening: Monique Vogelzang such as approving subject profiles and boards have enough freedom to develop II Interview: VSNU, VH, NRTO courses attended abroad, and issuing at a professional level. That means that we exemptions and diplomas, examination as a university must make enough time III Message from the minister boards were also assigned a new task: and capacity available for the performance IV Interview: ISO quality assurance for interim and final of these tasks – both to the examination Martin Kropff V Workshops examinations, as a consequence of the boards, and to the lecturers and examiners Rector magnificus, Wageningen UR Improved Governance (Higher Education) in the research groups.’ VI Interview: Hobéon, QANU, AeQui Act. I am proud to say that the examination VII Infomarket boards shouldered this responsibility in a Is a certain institutional culture required rigorous and dynamic fashion. In doing so in order for examination boards to VIII Presentation: Jan Anthonie Bruijn they considered which role they wish to play perform effectively? IX Panel discussion in the quality control cycle at Wageningen ‘Definitely, an open culture with short University in order to fulfil this responsi- lines of communication helps to initiate X Closing address: Monique Vogelzang ‘The examination bility. They have drawn up a testing and discussions on quality. But that same open Literature and links assessment policy that describes their culture must also be secured through assurance system. Research group visits clear and concise regulations. Clarity form part of this policy, which involve is important; too much bureaucracy is boards shouldered the examination board visiting a research seen as cumbersome. We have found an V.3 group to discuss the examinations with effective balance between an open culture all examiners. This is not only inspiring and clearly defined rules. The examination their responsibility to lecturers, but also provides them with board members act mostly on their own useful feedback on the design of their initiative, but you do need the right people exams.’ – people who have the organisation’s 10+ confidence. One crucial requirement is in a rigorous Can you describe the communication for the executive board and examination between the executive board and boards to work in tandem towards the same examination boards? goals.’ fashion’ ‘In their annual reports, the examination boards outline the activities they have Is there anything else you would like to carried out that are designed to evaluate the share with us? quality of interim and final examinations. ‘I would like to stress how important the programmes, delivering well-educated They also advise the executive board on theme of today’s conference is. I will be students. Wageningen University aims MORE INFORMATION matters that warrant its attention in this leaving for Mexico soon, but I am happy to to be the best. For years now we have respect. One such matter concerns whether be here today. We must all work together received high scores for 21 of our 28 MSc Rules and Regulations the current capacity of examination boards to reach our goal of ensuring high-quality programmes, and our BSc programmes is still adequate, given the rising student education, and the Inspectorate plays an perform well too. The examination boards Testing and Assessment Policy population. Based on the annual report, I important part in this process. The existing play a significant part in this achievement, hold discussions with the four chairpersons. regulations and quality assurance systems ensuring (among other things) that the Workshop Presentation Ensuring the quality of interim and final enable the provision of quality study quality of our programmes remains high.’

Further Improvement a coser look 20 Workshops Quality quantified: the building-blocks of a (centralised) examination board

Martine Pieters, Justus examination themselves, how can anybody RELEVANCE TO ‘FURTHER IMPROVEMENT’ Tengbergen and Nico Scheeres know whether the result is an accurate The report states that most examination boards Foreword representation of the student’s level? NCOI monitor the quality of testing by investigating the (NCOI) and NTI have set up a rigorous AO/IC system quality of examinations, ascertaining whether I Opening: Monique Vogelzang to address this very problem, as Pieters and guidelines are being followed, and similar means. II Interview: VSNU, VH, NRTO Examination boards are responsible for Tengbergen explained. This workshop examined the administrative or- the quality of examinations. However, ganisation and internal control (AO/IC) necessary III Message from the minister in the Higher Education and Research Reports and standards in order for examination boards to provide effec- IV Interview: ISO Act (WHW), one of the most important The two universities of applied sciences tive supervision. V Workshops aspects of examination – determining the have defined separate roles for the activities results – is delegated to the examiner. So of teaching, examination design and VI Interview: Hobéon, QANU, AeQui how can examination boards ensure they examination assessment (i.e. segregation VII Infomarket are always in control? Martine Pieters, of duties) and documented their processes Justus Tengbergen and Nico Scheeres (ISO 9001). They also have a clear reporting VIII Presentation: Jan Anthonie Bruijn outlined how the NCOI and NTI Univer- structure that gives the examination board IX Panel discussion sities of Applied Sciences deal with this insight into existing and potential risks. The dilemma. reports provide information on the quality X Closing address: Monique Vogelzang of examinations and the performance of Literature and links Effective administrative organisation examiners, and identify potentially weaker and internal control (AO/IC) lays the students. Comparison against standards foundations for an examination board (including bandwidths) enables the that performs well, and is also essential to examination board to take prompt, targeted collaboration between the examination action whenever necessary, resulting in an V.4 board and the exam administration. examination board that is genuinely ‘in The basic components of AO/IC include control’. legislation, the integrity of all parties concerned, segregation of control duties, 10+ manageable processes (including risk management) and monitoring. Although Pieters and Tengbergen devoted attention ‘A clear reporting structure to all AO/IC components, they focused on the last three, which are key factors in determining the extent to which an examination board is in control. gives the examination board

The institution’s executive board is MORE INFORMATION responsible for structuring and supporting insight into existing and the entire AO/IC; the examination board NCOI reports is responsible for its operation. One complicating element is the fact that the Workshop Presentation examination board does not itself carry out potential risks’ all the activities necessary for remaining in control. Without personally assessing an

Further Improvement a coser look 21 Workshops

Further Improvement a coser look 22 Workshops

‘I’m mainly here just to listen, ‘I actually have the day off today, I’d like to learn how to measure but I’m still glad to be here. It’s Foreword quality. Sure, everybody has educational and fun. I’m glad I Opening: Monique Vogelzang opinions about quality, but I that the Inspectorate (that is, the II Interview: VSNU, VH, NRTO doubt whether such opinions are government) has finally taken the III Message from the minister always objective. I mean, surely initiative to get all of the examina- IV Interview: ISO it’s important to know what a tion boards together in the V Workshops diploma is worth, and how well spirit of enthusiasm, rather than VI Interview: Hobéon, QANU, AeQui a student has performed? Of supervision. I think it’s important course you can assess this using for the Inspectorate to provide VII Infomarket examinations, but how do you opportunities for networking and VIII Presentation: Jan Anthonie Bruijn measure the quality of the exam- knowledge exchange. We all need IX Panel discussion ination itself? On the other hand, to collaborate more effectively to X Closing address: Monique Vogelzang perhaps we are trying to measure improve. Today, I hope to learn Literature and links and quantify too much, but that’s more from my colleagues about a completely different kettle of the precarious balance between fish.’ what we want to achieve, and the V.4V resources and support we need Siert Knigge for that. Networking is also an External Chairman of the Central important element, which the Examination Board for Zeeland workshops cater for very well. 10+ University of Applied Sciences Hopefully they’ll organise more of these events, a shindig like this once a year seems like a good idea to me. I think that the fact it is happening at all is a success.’

Frans de Swart Chairman of the NTI Examination Board & member of the Inholland Graduation Committee

Further Improvement a coser look 23 Workshops Examination boards: A question of culture?

Sibe Doosje and Marie-Jet Fennema This approach also applies to examination RELEVANCE TO ‘FURTHER IMPROVEMENT’ (Utrecht University) boards. For example, the rector and the dean In its report, the Inspectorate advises all parties to Foreword hold an annual quality assurance meeting jointly decide on a clear and transparent position whose agenda includes the examination for examination boards within the context of their I Opening: Monique Vogelzang Do universities benefit from a single uniform boards’ performance; examination board institutions. This also means that the executive II Interview: VSNU, VH, NRTO quality assurance system, or is it better for a networks hold regular events; the university board must provide an organisational structure system to cater for the specialised nature and earmarks funds for the encouragement of and a quality assurance culture that recognises III Message from the minister culture within degree programmes? Utrecht innovation in education; and there is an and acknowledges the examination boards’ IV Interview: ISO University believes the latter. The Utrecht academic leadership training programme. importance. This workshop examined various internal quality assurance system describes V Workshops ways to encourage a quality assurance culture. the ‘what’ (i.e. the minimum requirements) Successful approach VI Interview: Hobéon, QANU, AeQui but not the ‘how’, and the university Utrecht University’s experience with promotes a culture of dialogue and mutual VII Infomarket examination boards shows that this approach learning. Marie-Jet Fennema and Sibe Doosje is successful. Despite the lack of prescribed VIII Presentation: Jan Anthonie Bruijn explained the approach taken by Utrecht procedures, examination boards have learned TIPS FOR A BETTER QUALITY CULTURE IX Panel discussion University. from each other’s practices – both good During the workshop, participants were and bad. In the course of time, they started asked what examination boards could do X Closing address: Monique Vogelzang The belief at Utrecht University is that quality operating in more or less the same way. Most to promote a quality culture. The respons- Literature and links education begins with a clear vision, which they boards, for example, now have a separate es yielded a number of useful suggestions: have documented in a University Directive that assessment quality committee that advises applies to all study programmes. The associated them on the quality of tests. -Start discussions and keep lines of quality assurance system allows plenty of communication open. freedom for local interpretation, based on the As an example of innovation, Sibe Doosje cited V.5 nature and culture of each faculty. Accordingly, the dialogue on the evaluation of multiple- -The examination board should give lecturers Utrecht University has no university-wide choice questions that has sprung up within the opportunity to submit an exam to them manuals for examination boards, although it the Faculty of Humanities. The MC+ project, before administering it to students. This will does produce guidebooks. with funding from the university’s education enable the examination board to evaluate the 10+ innovation budget, has put that dialogue in quality of the exam in advance. Shared responsibility the spotlight and accelerated its progress. No matter how good a quality assurance system The project has also led to the production of - Explain the importance of good-quality looks on paper, its effectiveness will hinge on a guidebook intended to increase the quality tests to lecturers, as well as the role played the right culture to sustain it, said Marie-Jet of multiple-choice testing, and has helped to by the examination board and what they can Fennema. A culture in which requesting and raise awareness of responsible testing methods expect from them. Communicate and advise. receiving feedback is a matter of course, where among examiners. MC+ has also generated there is continuous improvement, and one that more shared ownership of the quality of - Appoint somebody with legal and/or also aids the realisation that the system is part testing. This multiple-choice question approach mediation expertise as an external member MORE INFORMATION of a larger institutional whole, so that quality is expected to spread to other faculties. of the examination board. assurance becomes a truly shared responsi- Guide to MC tests bility. To create the desired quality assurance - Let the common good prevail, and make all culture, Utrecht University has established those involved aware of their role within the Workshop Presentation three key principles: monitoring and critical larger context of quality assurance. reflection, innovation, and fostering leadership UU Education Directive* development.

Further Improvement a coser look 24 *English translation Workshops

Further Improvement a coser look 25 Workshops Collaboration on test quality: rules of engagement

Els van Zijl and Mieke Jaspers (Fontys the role of ‘watchdog’ raises several questions. RELEVANCE TO ‘FURTHER IMPROVEMENT’ Universities of Applied Sciences) Who directs the examination board? Who is In its report, the Inspectorate advises all parties to Foreword responsible for the quality of testing? What is jointly decide on a clear and transparent position the role of testing and assessment experts, or a for examination boards within the context of I Opening: Monique Vogelzang Curriculum committees, testing and potential assessment committee? It is important their institutions. To aid this process, examination II Interview: VSNU, VH, NRTO assessment experts, examiners, adminis- that everybody’s role be clearly defined, and that boards, executive boards and examiners are trative staff, invigilators, timetablers... There they do not overstep the bounds of their authority advised to discuss how they wish to distribute the III Message from the minister are many bodies and officers besides the or expertise. Fontys has documented these responsibilities and authorisations. This workshop IV Interview: ISO examination board who are involved in testing principles in a number of guidelines (see the presented some potential guidelines to use when and examination. Chaos, ad-hoc solutions and interview on the following page). doing so. V Workshops the constant feeling of having to ‘reinvent the VI Interview: Hobéon, QANU, AeQui wheel’ are the results of a lack of systematic Van Zijl and Jaspers argued for a culture focused cooperation among the various parties. In VII Infomarket on development, in which professionals are and this workshop, Els van Zijl and Mieke Jaspers remain responsible for the quality of the tests. VIII Presentation: Jan Anthonie Bruijn discussed a number of strategies to help avoid The examination board alerts and advises the IX Panel discussion these situations. management on how to structure the organisation of testing to include adequate ‘inbuilt’ X Closing address: Monique Vogelzang Since 2010 and the introduction of the Improved guarantees, so that quality can always be ensured. Literature and links Governance (Higher Education) Act, it has been Van Zijl and Jaspers emphasised the fact that this the responsibility of examination boards to is primarily a question of structure: only this way ensure the quality of testing and examinations can the organisation grow to reduce the need – a significant and pro-active steering and for monitoring and interventions, on the part of V.6 monitoring role in the quality assurance system both examination boards and the management. 10+ for testing and examination. In practice, however,

MORE INFORMATION Collaboration Guide Advisory report Role of the examination board

Further Improvement a coser look 26 Workshops ‘We must move towards a culture focused on development, a culture that allows for mistakes to be made’ Els van Zijl and Mieke Jaspers (Fontys Universities of Applied Sciences)

Foreword What are the key rules when it comes to I Opening: Monique Vogelzang collaboration between faculty manage- Which of the above rules proves most II Interview: VSNU, VH, NRTO ment, the examination board, testing and difficult? assessment experts and the Fontys execu- ‘Acting in accordance with strictly defined III Message from the minister tive board? roles has proven difficult, because tasks are IV Interview: ISO ‘We stick to a number of rules that we have often poorly defined and responsibilities drawn up based on our experiences from re- are not covered effectively. This applies V Workshops cent years. The first key rule is for everybody in particular to assessment committees VI Interview: Hobéon, QANU, AeQui to adhere strictly to their role in the process, or testing and assessment experts, faculty according to clearly defined tasks. We have management and examination boards. We VII Infomarket noticed that a failure to properly delineate notice that we can cause confusion with VIII Presentation: Jan Anthonie Bruijn roles, duties and responsibilities can quickly questions such as: How would you define IX Panel discussion lead to annoyance, frustration and occa- “assurance”? Who should structure it? Who sionally even conflicts. Secondly, there must is responsible for testing and assessment X Closing address: Monique Vogelzang be regular consultation between the exec- policy and the test programme? Who draws Literature and links utive board and examination boards based these up? Who manages the testing and on a joint plan and thorough analyses. Ex- assessment experts? amination boards regularly report that their observations and analyses do not prompt The role and position of the assessment any action. Furthermore, expectations must committee in relation to the examination Els van Zijl Senior consultant, Education and Research V.6 be effectively managed, and appropriate to board tend to spark a lot of controversy. In Office, Fontys Universities of Applied the organisation’s current stage of develop- our opinion, testing and assessment experts Sciences ment. should mainly support the examiners. They should also contribute to the development 10+ A third rule that we would like to mention is of testing and assessment policy, test pro- that the examination board must not ap- grammes and other frameworks. Testing propriate any other tasks from the ‘chain’. and assessment experts therefore play a Some examination boards have trouble supporting role, on behalf of the executive with this: they do it because of their sense board or faculty management. The roles of responsibility, and because “otherwise and responsibilities of the key bodies and things will go wrong”. But it actually has employees are described in detail in our the opposite effect, because it stops the publication titled “Working together on organisation from developing further. the quality of testing in higher education” MORE INFORMATION One final (related) suggestion is that there (Samenwerken aan toetskwaliteit in het should be as little monitoring as possible. hoger onderwijs).’ Collaboration Guide Nobody likes supervision, or the “red pen”. The trick is to ensure adequate assurance What is the impact of following the above Advisory report Role of the throughout the chain. The professionals’ rules? responsibilities must be returned to them, ‘We have already mentioned several effects, Mieke Jaspers examination board Senior consultant, Education and Research and they must receive adequate facilitative such as the fact that professionals can once Office, Fontys Universities support from the faculty management. It is again take responsibility. Strict adherence to of Applied Sciences the task of the executive board and faculty detailed task descriptions is the >> management to effectively structure such >> first step towards a quality-oriented assurance.’ culture. The greatest impactFurther occurs when Improvement a coser look 27 the executive board and examination boards away. Drawing up a joint plan based on a thor- Workshops realise that they can benefit one another, ough analysis of the quality of testing resolved provided they stick to the same rules. In such the issue. cases, the examination board must have sufficient expertise and be clearly positioned as Another common occurrence is for examina- an independent body, while the faculty man- tion boards to check all tests. By once again agement retains responsibility for the quality making this component of the testing cycle of testing. The examination board is there to the examiners’ responsibility (under the su- help the management identify the strengths pervision of testing and assessment experts), and weaknesses of the test system, giving the quality improved noticeably and the pro- Foreword management a body that is able to provide fessionals entered into dialogues on quality active feedback on the structure of the “test standards.’ I Opening: Monique Vogelzang mechanism”. II Interview: VSNU, VH, NRTO Are these rules independent agreements, or III Message from the minister Could you name some examples you have do they assume a certain culture within an seen of clear improvement? organisation? IV Interview: ISO ‘We worked with a programme where it had ‘They require a culture focused on devel- V Workshops become commonplace for students to submit opment where people are allowed to make all complaints regarding tests directly to the mistakes, and where the structure of the test VI Interview: Hobéon, QANU, AeQui examination board, who then felt obliged to system and collaboration are subject to evalu- VII Infomarket respond to them all. Many resits and resub- ation. This is still quite rare.’ missions were granted, but nothing improved. VIII Presentation: Jan Anthonie Bruijn Only once the complaints were redirected IX Panel discussion to the responsible employees did anything X Closing address: Monique Vogelzang change. Another example concerns a conflict that was brewing between the examination Literature and links board and the faculty management. Members of the management felt that the examination board was handing them guidelines and V.6 expecting them to be implemented straight 10+ ‘Who manages the testing and assessment experts? Who is responsible for MORE INFORMATION testing quality? Tasks and Collaboration Guide Advisory report Role of the responsibilities should be examination board clearly defined’ Further Improvement a coser look 28 Workshops Quality assurance: from regulations to understanding

Foreword Edith Hooge and effective. Every morning, for exam- RELEVANCE TO ‘FURTHER IMPROVEMENT’ I Opening: Monique Vogelzang (TIAS, Tilburg University) ple, the Ministry of Education, Culture and In its report, the Inspectorate advises all parties II Interview: VSNU, VH, NRTO Science tweets an example of quality edu- to jointly decide on a clear and transparent po- cation, influencing perceptions and ideas sition for examination boards within the context III Message from the minister Although the education system has a long about quality education and creating a sup- of their institutions. To aid this process, examina- IV Interview: ISO history of regulations, there is also much port base. But if you go too far, the sword tion boards, executive boards and examiners are support for deregulation: those working will lose its edge: people will see through it. advised to discuss how they wish to distribute the V Workshops in education often express their dissatis- A variation on the three policy instruments responsibilities and authorisations. This workshop VI Interview: Hobéon, QANU, AeQui faction with all the rules. And it is certain- can have powerful effects: instead of start- presented some potential guidelines to use when ly not only the government who makes VII Infomarket ing with regulations, start by generating a doing so. rules: institutions also do this them- support base, and continue organising from VIII Presentation: Jan Anthonie Bruijn selves, including examination boards. there. Why such an enormous urge to regulate? IX Panel discussion It is the task of the examination board to And is there another way? Edith Hooge, Examination boards still have much to learn genuinely ensure the quality of tests and X Closing address: Monique Vogelzang Professor of Boards and Governance in in this area. The Further Improvement re- examinations, and to avoid turning things Education, provided a reflection on regu- port showed that they are now concentrat- Literature and links into a bureaucratic exercise. Two perspec- lation by examination boards. ing mostly on matters related to processes, tives can help in this respect: ‘from perfor- and little on the content of tests. This may mance to mastery orientation’, and combin- Examination boards direct their focus at be because it is not possible to regulate ing hard and soft information processing. the heart of education: testing and ex- content – lecturers, after all, require pro- These perspectives can help examination V.7 amination. In doing so, they often create fessional freedom. But what does this say boards to resist the urge to regulate, and to regulations. But why? Because we feel un- about quality assurance for the content of fulfil their responsibilities in other ways. comfortable when faced with inequality and tests and examinations? Should the exam- differences. We want uniformity and stan- ination board not worry about this at all? No recipe for education dardisation, otherwise the results could be How can the task of supervision be defined 10+ Is it even possible to fully regulate edu- unintended. Plus, a lack of rules means less to avoid creating a body focused exclusively cation? No. The job still requires human security: people must then make their own on processes? decisions. Rules provide comfort. Neverthe- input, and it is dangerous to just impose all sorts of rules and regulations. Baking a less, we regularly discover that regulations A bureaucratic exercise? cake is easy: just use a recipe. But raising are not compatible with the nature of ed- Currently, regulations often play a key role and educating children is more complex, ucation, and that they usually have a detri- in the supervisory activities of examination and formulas, protocols and codes are not mental effect. boards. These regulations can have varying enough. effects. The frameworks, guidebooks and MORE INFORMATION Three families guidelines that govern the work of examin- Regulations are an instrument of policy. ers and lecturers who create tests are some- Essay by Edith Hooge There are three families of policy instru- times very effective, and sometimes less so. ments: in addition to the ‘whips’ (laws and Regulations may be accepted, but they can Further Improvement* regulations), there are the ‘carrots’ (finan- also elicit other behaviours: people may cial incentives or sanctions) and the ‘ser- start negotiating and try to reach compro- Workshop Presentation mons’ (persuading people and establishing mises, to evade regulations or to defy them. frameworks). This last method is modern We call this ‘gaming’.

Further Improvement a coser look 29 *English translation Workshops

Further Improvement a coser look 30 Workshops ‘Examination boards can’t do everything from behind a desk’ Edith Hooge (TIAS, Tilburg University)

Can examination boards perform effectively without formulating guidelines for Foreword examiners? ‘Examination boards are there to ensure I Opening: Monique Vogelzang high-quality tests and examination. Simply II Interview: VSNU, VH, NRTO establishing regulations is not enough: en- couraging other parties to collaborate and III Message from the minister ‘Don’t just to exchange information to promote the IV Interview: ISO creation of quality examinations is a major V Workshops part of this process. Examination boards therefore shouldn’t just spew out regula- spew out VI Interview: Hobéon, QANU, AeQui tions, but should allow for the professional VII Infomarket freedom of teaching staff. It is the job of examination boards to verify whether the VIII Presentation: Jan Anthonie Bruijn regulations, quality of testing is sufficient, however.’ IX Panel discussion You state that regulation can go too far, X Closing address: Monique Vogelzang turning quality assurance into a bureau- but allow Literature and links cratic exercise. How can examination boards avoid this? Edith Hooge ‘It takes discipline. Examination boards for the Professor of Boards and Governance in must not give in to regulation as a knee-jerk Education, TIAS, Tilburg University reaction, but should ask themselves wheth- V.7 er any new regulations are really necessary, or whether the situation requires a different professional approach. The board must also monitor the actual effects of measures and regulations, 10+ and check to see whether they are feasible freedom of in practice. This costs extra time and energy, because people need to maintain contact and enter into discussions with lecturers, teaching staff’ faculty managers and students. Nobody can do that from behind a desk. It also helps if members of the examination board have subject-related expertise.’ MORE INFORMATION Essay by Edith Hooge Further Improvement* Workshop Presentation

Further Improvement a coser look 31 *English translation Workshops The examination board through the eyes of a student

Klaasjan Boon (LSVb) • Precision Many students are uncertain about deadlines or response times, creating Foreword Students are at the heart of the work the impression that examination boards performed by examination boards. But I Opening: Monique Vogelzang are careless whenever they announce a how do students view the examination decision later than expected. II Interview: VSNU, VH, NRTO boards? What do they know about the III Message from the minister boards’ work, and how do they regard The workshop participants continued it? In December 2014 the Dutch National IV Interview: ISO by thinking of innovative solutions to Union of Students (LSVb) published the above-mentioned problems, and V Workshops a study on the role of examination exchanged ideas and good practices for RELEVANCE TO ‘FURTHER IMPROVEMENT’ boards in higher education. The study VI Interview: Hobéon, QANU, AeQui each of the areas highlighted by Boon (see In the inquiry by the Inspectorate, 94% of exam- showed that students’ perceptions box on next page). Of course there is no ination boards state that students are informed VII Infomarket of examination boards do not always ‘cure-all’ that can prevent or solve every about the examination board’s tasks, authorisa- correspond to reality. Klaasjan Boon, a VIII Presentation: Jan Anthonie Bruijn problem, and a combination of several tions and procedures. Still, examination boards member of the LSVb Executive Board, solutions is desirable. One challenge for often exhibit limited transparency, and the Dutch IX Panel discussion highlighted the problem areas. examination boards is to think about what National Union of Students (LSVb) and the Dutch X Closing address: Monique Vogelzang they themselves could improve with regard National Student Association (ISO) believe that Boon discussed three areas where Literature and links to these areas, and to gain inspiration from students are often underinformed. The LSVb con- examination boards can make drastic the suggestions in the box overleaf. ducted its own study on the subject, and this work- improvements, as shown by the LSVb study: shop covered the topic from a student perspective. • Transparency V.8 It is often unclear to students exactly what goes on in the examination board, and why certain decisions are made. Students are also unaware of who the 10+ board members are. ‘Many students are • Independence Students who are familiar with the examination board notice that lecturers uncertain about deadlines, make decisions regarding their own subjects. What they do not see, however, is that lecturers virtually always leave the meeting whenever their own subjects creating the impression that are discussed, giving students the MORE INFORMATION impression that examination boards do LSVb study not always act fully independently. examination boards are careless’

Further Improvement a coser look 32 GOOD PRACTICES AND TIPS FOR EFFECTIVE COMMUNICATION WITH STUDENTS Workshops What can examination boards do to improve their transparency, independence and precision in order to more effectively meet the needs of students? The suggestions by workshop participants for each category are summarised and presented below.

Transparency Independence Precision • If a student sends a request to the • The examination board could take on a • Explaining all of the considerations when examination board that does not belong student as an (acting) member, in order issuing a decision will inform students of there, the board itself could forward it to to ensure independence and include a why their request has been granted or the correct body and inform the student student perspective. declined. When doing so, it is important of what has happened. This way, students to remain brief and succinct, so as not to Foreword are prevented from feeling as though they • Periodic consultation between the burden students with too much text. I Opening: Monique Vogelzang are being given the run-around. examination board and the programme committee would not only promote • Creating a concise, clear brochure II Interview: VSNU, VH, NRTO • The examination board could introduce independence, but would also benefit the outlining the examination board’s tasks III Message from the minister itself to the students at the start of the mutual exchange of information and the and authorisations will raise student IV Interview: ISO study programme. Although they could coordination of tasks and activities. awareness of the examination board and do so the start of the year, this is a time (better) inform them of deadlines and V Workshops when students are already inundated with • Placing an FAQ on the faculty website response times. VI Interview: Hobéon, QANU, AeQui information. The second study period is would enable the examination board to therefore probably a better idea. respond to common questions regarding • Offering to explain a decision face-to-face VII Infomarket its independence, tasks and authori- will enable examination boards to inform VIII Presentation: Jan Anthonie Bruijn • By asking students about their sations. students of the relevant considerations in experiences/satisfaction following a personal fashion.. IX Panel discussion contact with the examination board, • In addition to an FAQ, the decision tree X Closing address: Monique Vogelzang the board demonstrates its concern and could also be put on the website to inform • Informing students via email instead of by Literature and links also receives information on possible students of which types of questions the post will speed up contact. improvement areas. examination board will handle, and how they are dealt with. • If the examination board maintains • Attending information days and events effective contact with academic will enable the examination board to • Appointing an external member would counsellors, clearly coordinating tasks and V.8 establish a presence among future improve the board’s independent image. authorisations with them, students will students as an important decision-making be referred less often to the examination body. • Clearly stating the available appeal board for matters that they should take options at the end of each decision gives elsewhere. 10+ • Holding a (possibly weekly) walk-in office students an honest and independent hour will increase the examination board’s impression of the examination board. visibility and give it a human face, also making it more accessible and personal.

• The examination board could draw up a set of standing orders, so that students know which procedures and regulations are used. MORE INFORMATION LSVb study

Further Improvement a coser look 33 Workshops The examination board as a professional community

John Huizinga (TIAS, Utrecht The objective of the Improved Governance RELEVANCE TO ‘FURTHER IMPROVEMENT’ University of Applied Sciences) (Higher Education) Act is to enable higher In its report, the Inspectorate advises all parties to Foreword education examination boards to proper- jointly decide on a clear and transparent position and Frans de Vijlder (HAN ly fulfil their responsibilities using what for examination boards within the context of their I Opening: Monique Vogelzang University of Applied Sciences) is called ‘professionals governance’. The institutions. The Inspectorate also advises sharing II Interview: VSNU, VH, NRTO formation of networks and mutual ex- knowledge via national networks. This workshop change of knowledge are key aspects of this looked at these two matters. III Message from the minister Examination boards bear a major responsibility, which has only increased since approach. The search for the right way for IV Interview: ISO the introduction of the Improved Gover- examination boards to fulfil their role is a V Workshops nance (Higher Education) Act. Examina- cyclical learning process, said Huizinga and De Vijlder. VI Interview: Hobéon, QANU, AeQui tion boards are now charged with the task of ensuring the quality of interim and VII Infomarket final examinations much more explicitly They also discussed the development of digital research networks that can contribute VIII Presentation: Jan Anthonie Bruijn than before. How can they fulfil this responsibility, and what is their relation- to the emergence of a professional commu- IX Panel discussion ship with the executive board and faculty nity, which performs its duties based on a pool of knowledge that has been academi- X Closing address: Monique Vogelzang management? John Huizinga and Frans de Vijlder considered these aspects. cally validated to the greatest extent possi- Literature and links ble. Knowledge circulation is crucial in this respect. 10+V.9

Further Improvement a coser look 34 Workshops

Further Improvement a coser look 35 Workshops

‘I work at the Dutch National ‘Examination boards are in a state Student Association (ISO). Since of flux right now, so I’m hoping Foreword there is so much talk about stu- to gather some information on I Opening: Monique Vogelzang dents here, it’s nice for there to new developments. Interacting II Interview: VSNU, VH, NRTO be a few of us around. Today the with others, gaining inspiration III Message from the minister overwhelming focus is on exam- and networking are also on my IV Interview: ISO ination boards, but it’s great to be list of important things to do. My V Workshops here and to make it clear that the main interest lies with ensuring VI Interview: Hobéon, QANU, AeQui students mustn’t be forgotten. the quality of examiners, also as Right now, I myself am working part of the recent developments VII Infomarket on a study concentrating on test- in basic and senior examination VIII Presentation: Jan Anthonie Bruijn ing, and on feedback in particu- qualifications (BKE and SKE). IX Panel discussion lar. Examination boards already There was a nice discussion at the X Closing address: Monique Vogelzang devote a lot of thought to testing Education Café about the core Literature and links and committees, but the feedback duties of examination boards – they receive from students could it’s good to hear that there are be put to better use. I hope to others who also wrestle with the V.8V put forward that message today. I same issues, such as students want to sound the students’ trum- studying abroad.’ pet, and see the boards respond by saying that they want to do Bertus Schokker 10+ more with student feedback.’ Examination Board member, Stenden University of Applied Lisanne van Kessel Sciences Project officer, Dutch National Student Association (ISO)

Further Improvement a coser look 36 Workshops Tests, examination and the role of examination boards in higher education accreditation

Foreword I Opening: Monique Vogelzang II Interview: VSNU, VH, NRTO III Message from the minister Wienke Blomen, Rob Stapert and In an accreditation inspection, however, the RELEVANCE TO ‘FURTHER IMPROVEMENT’ examination board’s ‘watchful eye’ is the In its report, the Inspectorate notes that IV Interview: ISO Frank Hendriks (Hobéon) key focus. Hobéon’s development-oriented decisions by the Accreditation Organisation of V Workshops approach takes this into consideration. In the Netherlands and Flanders (NVAO) and the The role of testing and examination in the VI Interview: Hobéon, QANU, AeQui such cases, the inspection panel looks at underlying observations from inspection reports accreditation of higher-education pro- how the examination board organises the correspond to the results of the inquiry. Nearly all VII Infomarket grammes has clearly become more prom- quality assurance of examinations (testing faculties currently undergoing a recovery process inent. But what exactly are inspection VIII Presentation: Jan Anthonie Bruijn and assessment), with a clear emphasis on report problems with the examination board. This panels looking for in the performance of whether the board is aware of its own stage workshop outlined how inspection panels from IX Panel discussion an examination board? This workshop of development. Does it know what is going Hobéon view the performance of examination X Closing address: Monique Vogelzang run by Hobéon revealed that evaluations well, and what it could (or should) do bet- boards. concentrate primarily on proportionality ter? Literature and links and attention to the examination board’s stage of development.

An inspection will of course look at the V.10 formal position occupied by examination THREE KEY QUESTIONS boards, and at the support they receive: is The discussion between an inspection panel and an examination board can be the board properly equipped to fulfil its role reduced to three key questions: effectively? In addition to discussions with the examination board, the management, • What has the examination board done over the previous year/until now to improve the 10+ teaching staff and students, the panel will quality of interim and final examinations (testing and assessment)? In other words: what also perform its evaluation using relevant guidelines and instructions have been issued to clearly define the intended quality? underlying documentation, such as annual plans and reports. The panel will then in- • What has the examination board done over the past year to gain a clear idea of vestigate whether formal matters and duties the actual quality of the structure and implementation, and what was the result? And within the examination board are finalised subsequently: what proposals has the examination board made for further development and in order. or improvement?

• Where will the examination board be in 2016? What is on the agenda for the year ahead, MORE INFORMATION and what are the priorities? Examination board: background Rescinding appointments Interview with Blomen, et al.

Further Improvement a coser look 37 Workshops How can institutions work on the testing and assessment competence of their examiners?

Remko van der Lei and Brenda members. The programme is structured as RELEVANCE TO ‘FURTHER IMPROVEMENT’ Aalders (Hanze University of follows: The examiners all start with one of In its report, the Inspectorate stresses the their own exams. They receive a workbook importance of testing and assessment expertise. Foreword Applied Sciences, Groningen) containing an empty portfolio, a bibliogra- Examiners require further professional I Opening: Monique Vogelzang phy and a checklist which they must use to development. In this workshop, Hanze University of Applied Sciences in Groningen demonstrates its II Interview: VSNU, VH, NRTO In response to the External Validation collate all of the documentation relevant to Report, the Netherlands Association of the creation of the test. They study the liter- own approach to professional development. III Message from the minister Universities of Applied Sciences (VH) has ature and reflect on their test, then attend a IV Interview: ISO drawn up a schedule of requirements de- criterion-based interview on their portfolio, scribing the basic learning outcomes for which interview is evaluated. WHAT IS TESTING AND ASSESSMENT COMPETENCE V Workshops examiners, which are attested via the Basic ALL ABOUT? VI Interview: Hobéon, QANU, AeQui Examination Qualification (Basiskwalifi- According to Van der Lei and Aalders, BKE According to the participants, testing and assessment catie examinering, or BKE). Hanze Univer- certification is designed to improve the ba- VII Infomarket expertise involves: sity of Applied Sciences in Groningen tests sic standard of examiners across the entire VIII Presentation: Jan Anthonie Bruijn its examiners for the BKE using a port- testing spectrum, making it a relevant addi- • knowledge of the institution’s testing and assessment IX Panel discussion folio and a criterion-based interview. In tion to the palette of professional activities policy; this practical session, Remko van der Lei among (even experienced) lecturer-exam- • an awareness of the position of the test within the larger X Closing address: Monique Vogelzang and Brenda Aalders invited participants iners in higher education. Examiners must testing and assessment plan; Literature and links to contribute their ideas regarding this demonstrate their testing and assessment • knowledge of forms of testing and the ability to apply method of ensuring the quality of tests. competence at all stages of the testing cycle. them correctly and at the right time (e.g. when to use The advent of the BKE and its senior equiva- formative/summative tests); Van der Lei and Aalders first asked the par- lent (the SKE) also has a positive impact on • relevant subject knowledge; ticipants a few questions regarding testing assessment committees, changing their role • selectiveness of the test (will those who do not V.11 and assessment competence, the quality of from overseer to discussion partner. understand actually fail?); examiners and the potential role of the BKE • a precise formulation of learning objectives as a basis for in improving the quality of tests. In pairs, Since the BKE programme’s inception at the test; the participants shared their knowledge and Hanze University, student satisfaction re- • consistency between teaching and testing; 10+ experiences on these subjects (see box). Van garding testing has increased. Lecturers who • tests that are internally consistent, contain clear questions der Lei and Aalders then used the group’s have obtained their BKE certification are also and are worded effectively; feedback to establish connections with the pleased with the results, and a number of as- • intersubjectivity in the creation and evaluation of tests; theoretical framework of the test quality pyr- sessment committee and examination board • tests that are not (or not only) used as a means to ‘sort the amid within educational organisations (see members have now also obtained their SKE. wheat from the chaff’, but can also be used as a means of handout). The old adage of ‘every test is only providing feedback and enhancing learning. as strong as its weakest link’ applies to both Flywheel effect the pyramid and the associated testing cycle. The workshop participants were curious about the organisation’s response to the im- shared their enthusiasm, people’s openness MORE INFORMATION Externally validated BKE programmes plementation of the BKE programme. They increased, along with mutual discussion The workshop facilitators then discussed themselves regularly encounter resistance, about testing issues, and the willingness and Article: Van der Lei Hanze University’s experiences with the Ba- e.g. from their more senior colleagues who, eagerness among others to obtain the qual- sic Examination Qualification (BKE). Hanze with all their experience, fail to see the need ification too. External Validation Report is the only university of applied sciences for a BKE. Van der Lei and Aalders reported with an externally validated BKE programme that, at Hanze University, the training pro- If you are interested in the BKE/SKE pro- Responsible testing (by Fontys and Zuyd University of Applied gramme had created a kind of flywheel ef- gramme at Hanze University of Applied Sciences, please contact Remko van der Lei: Workshop handout* Sciences) and tests all of its 1600 examiners. fect. Plenty of energy was invested in the BKE BKE certification is required for all assess- programme at the outset. As more and more [email protected]. ment committee and examination board lecturers completed the programme and Further Improvement a coser look 38 *English translation Workshops Calibration of final assignments in higher education

Marlies van Beek (Cito) RELEVANCE TO ‘FURTHER IMPROVEMENT’ In its report, the Inspectorate looks at examination boards’ responsibility to ensure the quality of Foreword Peer review can help promote the profes- examinations. This responsibility also involves sional development of examiners. One I Opening: Monique Vogelzang further professional development among form of peer review is the ‘calibration examiners. This workshop discussed one specific II Interview: VSNU, VH, NRTO session’, used in the assessment of form of development: calibration sessions. III Message from the minister final assignments or theses in order to increase the reliability of assessment. IV Interview: ISO During the workshop, Marlies van Beek, V Workshops trainer/consultant at Cito, showed what a VI Interview: Hobéon, QANU, AeQui calibration session entails.

VII Infomarket Calibration addresses various questions. WHAT HAPPENS DURING A VIII Presentation: Jan Anthonie Bruijn What do the assessment criteria actually CALIBRATION SESSION? mean? When is a satisfactory/unsatisfactory One way to organise a calibration session is IX Panel discussion mark awarded for each criterion? What as follows: X Closing address: Monique Vogelzang explanation (if any) is required for each criterion? Van Beek presented an exercise Literature and links 1. Examiners use the assessment model to show how a calibration session can be to assess a final assignment or thesis used to answer these questions. A guide independently of one another. to calibration sessions has been included 2. During the first meeting, the examiners in the final report by the Protocol expert discuss the assessments, facilitated by a V.12 group, which conducted research into a testing and assessment expert. Where do the joint protocol for the assessment of final/ similarities lie, and what are the differences? other assignments. Using this analysis the group then jointly develops a communal reference framework, 10+ improving the assessment form/process where necessary. The examiners then independently assess another final assignment or thesis, using the improved form/process. 3. Once these assessments have been analysed via a report, an evaluation session is held.

MORE INFORMATION Cito Calibrations Guide

Further Improvement a coser look 39 Workshops

Further Improvement a coser look 40 Workshops The interpretation of the Higher Education and Research Act (WHW) in relation to examination boards

Patrick Leushuis and Trees Rui- declare an examination invalid. It is permis- RELEVANCE TO ‘FURTHER IMPROVEMENT’ jgrok (Ministry of Education, Cul- sible, said Ruijgrok, if fraud or irregularities The report shows that examination boards are Foreword are discovered in the work of one or more currently struggling to interpret some of the stat- ture and Science, OCW) I Opening: Monique Vogelzang students. But as luck would have it, one of utory provisions. A common view of the right in- the attendees had just come from a work- terpretation is desirable, and the Inspectorate also II Interview: VSNU, VH, NRTO The position of examination boards in shop where the same question had been outlines how new developments (such as the plans III Message from the minister the Higher Education and Research Act asked, and the answer was ‘no’. Discussion to make higher education more flexible) are in- (WHW) was redefined in 2010. However, ensued, and it quickly became clear that creasing the need for further improvement among IV Interview: ISO examination boards are not always clear the interpretation depends on the specific examination boards. This workshop focused on V Workshops on how they should interpret the statuto- circumstances of the case. The examination both of these topics. ry provisions. Patrick Leushuis and Trees board may declare an examination inval- VI Interview: Hobéon, QANU, AeQui Ruigrok, both senior policy officers at the id, but cannot alter any grades awarded by VII Infomarket Higher Education and Study Financing the examiners. The tasks of examination VIII Presentation: Jan Anthonie Bruijn department of the Ministry of Education, assessment and grading are the examin- Culture and Science (OCW), discussed sev- er’s responsibility under the WHW, and no IX Panel discussion eral interpretation issues with the partic- provisions in the Teaching and Examination ‘The X Closing address: Monique Vogelzang ipants in this workshop. They also looked Regulations may deviate from this princi- at two other important matters concern- ple. Literature and links ing examination boards: the pilots in programme flexibility, and – as a logical Own investigation? examination extension thereof – the experiments in A third question was raised as follows: demand-based financing. ‘What is the meaning of the WHW regula- V.13 tion that authorises the examination board The interpretation of the Higher Educa- to conduct its own independent investi- board may tion and Research Act gation following all examinations?’ This The Further Improvement* report deals regulation does not pertain to an investi- with several issues related to the interpreta- gation into the quality of examinations, declare an 10+ tion of statutory provisions. The first part of Ruijgrok explained, but rather to students’ the workshop was devoted to these issues. knowledge, understanding and skills. These types of investigations must be listed in examination Can an examiner’s appointment be revoked? the Teaching and Examination Regulations The participants made short work of the (OER) as part of the examination. All stu- first question: ‘Can an examiner’s appoint- dents are subject to such an investigation, ment be revoked if they fail to perform which must take the same form for each invalid, but adequately?’ The answer is already included student. Because it is a final examination in the Further Improvement report: yes, and not an interim examination, exemp- MORE INFORMATION examination boards are authorised to do so. tions are not permitted. cannot alter any Further Improvement letter However, examiners must first be given the chance to improve their performance. Unanimity is not always possible Sections of the WHW The attendees certainly gave the ministry’s Can examinations be declared invalid? representatives a run for their money: all grades awarded’ Increased Flexibility letter One question requiring more time was that manner of exceptions were brought up that of whether the examination board may push the existing frameworks and >> Advisory report on flexibility Workshop Presentation Further Improvement Further Improvement* a coser look 41 *English translation Workshops ‘Examination is, and will remain, a job done by

>> regulations to the limit. For example: a impact examination boards. student was supposed to have been awarded a Foreword score of 1 for a certain subject, but received a Experiments in demand-based financing 6 due to a computer error. This was not dis- In the experiments in demand-based financ- I Opening: Monique Vogelzang covered until nine months later. What to do? ing, funding is provided by means of vouchers II Interview: VSNU, VH, NRTO What is clear is that examination is, and will which students can exchange for participation remain, a job done by human beings, Ruijgrok in accredited study programmes (or com- III Message from the minister concluded. It is certainly not always possible ponents thereof ) at either public or private IV Interview: ISO to provide an unambiguous answer to all institutions. The experiments apply to select- V Workshops questions that may arise. ed programmes in the engineering, healthcare and welfare sectors, which may recruit and VI Interview: Hobéon, QANU, AeQui Flexibility pilots enrol participants for phased, modular partic- VII Infomarket Part 2 of the workshop was devoted to the ipation in accredited programmes. The stipu- pilots for flexibility in adult higher education lations regarding a fixed campus are relaxed, VIII Presentation: Jan Anthonie Bruijn and experiments in demand-based financing. giving institutions the freedom to conduct IX Panel discussion The flexibility pilots are intended to create teaching activities on location. opportunities for making the design of part- X Closing address: Monique Vogelzang time and work-study programmes more flexi- Literature and links ble and attractive. Faculties can create tailored study paths by working with units comprising learning outcomes rather than with fixed units of teaching, and by including validation of students’ prior knowledge and skills V.13 so that the paths can be shortened. Using the workplace as a learning site and intensifying the use of online teaching methods allows more efficient access to study programmes by 10+ adults. The responses clearly reflected a great need to make education more flexible, and many representatives from universities of applied sciences expressed their eagerness to take part in the pilots. It was also pointed out that these developments are not only important to adults, but to all students in higher education. MORE INFORMATION In the interview on the following page, Le- ushuis explains how increased flexibility will Further Improvement letter Sections of the WHW Increased Flexibility letter Advisory report on flexibility Workshop Presentation Further Improvement Further Improvement* a coser look 42 *English translation Workshops ‘Learn from each other, seek each other out, ask each other questions’ Interview with Patrick Leushuis (Ministry of Education, Culture and Science, OCW)

The learning track-independent Another future change involves the assessment currently being trialled in the recognition of students’ prior learning Foreword pilots is a major innovation. What will outside the faculty. What role will this mean for examination boards? examination boards play here? I Opening: Monique Vogelzang ‘The flexibility pilots are abandoning ‘Many working adults have already gained II Interview: VSNU, VH, NRTO the one-size-fits-all approach based on relevant knowledge, understanding and a single curriculum that applies equally skills as part of their jobs, or through III Message from the minister to everybody. A diverse range of flexible various training courses. We want these IV Interview: ISO learning pathways can exist within the same results to be recognised and converted into V Workshops study programme. Nevertheless, there must course credits, so that study pathways can still be a guarantee that all of the various be designed to avoid teaching people things VI Interview: Hobéon, QANU, AeQui learning tracks lead to comparable learning they already know. Recognising the relevant Patrick Leushuis VII Infomarket outcomes, and that every diploma is worth results requires the application of proper Directorate of Higher Education and what it says it is: the achievement of the validation procedures and instruments, Study Finance Ministry of Education, VIII Presentation: Jan Anthonie Bruijn exit qualifications, in terms of both content based on the formulated learning outcomes Culture and Science IX Panel discussion and standards. This is why we apply an of the study programme. Next, examination assessment method that is independent of boards must be able to justify any decision X Closing address: Monique Vogelzang any learning track: the same criteria apply to to award course credits (or even diplomas) Literature and links everybody, regardless of the path they take. to candidates who demonstrably meet ‘Discuss with Examination boards will need to develop all of the requirements at exit level. guidelines that match both the specific Such decisions must be well-founded character of learning track-independent and documented so that the process is assessment and the associated customary transparent and traceable, and to ensure your fellows what V.13 testing and assessment methods. Then they accountability on the part of examination will need to make sure these are properly boards.’ monitored.’ works and what How can examination boards adapt to the 10+ How will this be implemented in new conditions? practice? ‘The flexibility pilots will be supported and doesn’t’ ‘During the workshop, examples were monitored by the Inspectorate, the Accred- cited such as progressive testing, portfolio itation Organisation of the Netherlands assessments and assessment centres, which and Flanders and the Ministry of Education, a number of institutions already have Culture and Science. Possible resources have relevant experience. Identify good experience with. The actual formulation include a guidebook containing guidelines, practices, discuss with your fellows what of the learning outcomes will be a crucial tips, effective examples, etc. for working works and what doesn’t, and don’t be afraid success factor, and will also have an impact with learning outcomes, learning track-in- to ask questions whenever you encounter MORE INFORMATION on the design of the learning track-inde- dependent assessment, recognition of dilemmas and issues in development or pendent assessment. It would therefore prior learning and quality assurance in implementation. Bring up matters that you Further Improvement letter seem obvious to involve examination work-study programmes. Events facilitating think you will benefit from discussion with boards in the creation of guidelines and knowledge exchange will also be organised, members of staff at other institutions – Sections of the WHW frameworks for the formulation and which is also general good advice: learn not just examination boards, but anybody Increased Flexibility letter adoption of the various units and learning from each other, seek each other out, visit within the organisation involved with the outcomes.’ other institutions and faculties that already flexibility pilots.’ Advisory report on flexibility Workshop Presentation Further Improvement Further Improvement* a coser look 43 *English translation Workshops The examination board in the legal precedents of the Higher Education Appeals Tribunal

Willem Beijk, Jan Nijenhof and RELEVANCE TO ‘FURTHER IMPROVEMENT’ Ben Olivier (Higher Education Examination boards are currently struggling to Foreword A document is available with legal precedent interpret some statutory provisions. Students are Appeals Tribunal, CBHO) information for examination boards, covering entitled to appeal to the Higher Education Appeals I Opening: Monique Vogelzang the following topics: Tribunal against decisions taken by examination II Interview: VSNU, VH, NRTO So what does the law actually say about boards, and it is important for examination boards fraud? How should examination boards 1. The examination board as a supervisory body to be aware of the relevant legal precedents. That III Message from the minister go about claims for compensation? was the subject of this workshop. IV Interview: ISO What are the regulations governing the 2. The examination board and examiners V Workshops composition of the examination board? In this workshop, the Higher Education 3. The examination board and exemptions VI Interview: Hobéon, QANU, AeQui Appeals Tribunal (CBHO) used the key VII Infomarket articles pertaining to examination boards 4. The examination board and its decisions in the Higher Education and Research Act VIII Presentation: Jan Anthonie Bruijn to clarify current legal precedents. 5. The examination board and informal ap- IX Panel discussion peals/complaints procedures Key topics discussed during the workshop X Closing address: Monique Vogelzang included the composition of examination 6. The examination board and settling out of Literature and links boards, the examination board as a party court to legal proceedings, and fraud and how it is dealt with by examination boards. 7. The examination board and fraud issues Informal complaints procedures were also discussed: is this practice desirable or not? 8. The examination board and the Iudicium V.14 The workshop facilitators also looked at the Abeundi (‘departure order’, Higher Education procedure for compensation. and Research Act, Section 7.42a)

9. The examination board as the implementing 10+ body of the BNSA 10. The examination board as a party to legal proceedings

11. The periodic penalty payment procedure

12. Compensation procedure (General Adminis- trative Law Act (Awb), Section 8:88 et seq.) MORE INFORMATION Legal precedents Sections of the WHW Workshop Presentation

Further Improvement a coser look 44 Workshops

Further Improvement a coser look 45 Workshops Valid and reliable testing in art education

Ton Lamers, Marcel Doorduin Three institutions shared their experienc- RELEVANCE TO ‘FURTHER IMPROVEMENT’ and Jackelien ter Burg (ArtEZ) es in four areas. In its report, the Inspectorate looks at examina- Foreword tion boards’ responsibility to ensure the quality of René de Klein and Anton Neg- Area 1: Guaranteeing exit levels (ArtEZ)In The examinations. and recommends that the boards I Opening: Monique Vogelzang gers (Fontys Universities of Ap- Higher Education and Research Act states engage in cross-institutional sharing of good II Interview: VSNU, VH, NRTO plied Sciences) that examination boards must use objec- practices. That is exactly what examination boards tive and expert methods to verify whether at institutions of art education did during these III Message from the minister Jan Wirken (Zuyd University of students fulfil the requirements set out in two workshops. IV Interview: ISO Applied Sciences) the Teaching and Examination Regulations V Workshops with regard to knowledge, understanding Examination procedures in art education and skills. This task has also been included VI Interview: Hobéon, QANU, AeQui differ from those at other faculties in a in the new accreditation framework of the VII Infomarket number of ways. Objective criteria can be Accreditation Organisation of the Nether- elusive in the arts, and in assessments, lands and Flanders (NVAO). In addition to VIII Presentation: Jan Anthonie Bruijn the feedback is often equally as import- the required system checks, this process IX Panel discussion ant as the grade itself. Arts programmes demands a subject-based assessment. In also work a lot with external assessors, far the arts, external assessors are often used X Closing address: Monique Vogelzang from all of whom have teaching qualifica- for this purpose, i.e. professionals who are Literature and links tions. So how do arts programmes ensure aware of the demands placed on artists and valid and reliable testing? art educators in the field. It is also important for teaching staff to cultivate a >> 10+V.15+16

MORE INFORMATION ArtEZ: Regulations ArtEZ: Policy AHK: Assessment Appointment examinors Format exam CM Further Improvement Workshop report a coser look 46 Workshops ‘The examination board can play a role in assuring feedback quality’ Foreword I Opening: Monique Vogelzang >> shared interpretation of the exit levels stip- Area 4: Finalising assessments (ArtEZ) ulated by the assessment criteria.. Performance feedback is essential to the fur- II Interview: VSNU, VH, NRTO ther development of all students in the arts. III Message from the minister Area 2: Assessment systems (Zuyd) The way in which assessments (i.e. grades) are The quality of tests is largely dependent on reached must also be made adequately clear IV Interview: ISO the test system used. Within any programme, to both students and other parties. The proper V Workshops the testing and assessment policy, programme justification of an assessment is an art in itself. VI Interview: Hobéon, QANU, AeQui and instruments must all be mutually coor- Peer review instruments are a useful means dinated. A proper testing system will fulfil all of improving feedback, and the examination VII Infomarket requirements pertaining to validity, reliability board can play a role in assuring feedback VIII Presentation: Jan Anthonie Bruijn and transparency. Describing and applying quality. the right testing criteria is not easy for many IX Panel discussion artistic disciplines, however, nor is it a simple A detailed report from the workshop is X Closing address: Monique Vogelzang matter to ensure quality throughout the entire included below. system, which is the examination board’s job. Literature and links In addition to assessing students, it is important therefore to also continually monitor the quality of tests.

V.15+16 Area 3: Quality of examiners (Fontys) The Act states that the examination board must appoint one or more examiners per test. These may be experts external to the institu- 10+ tion. Criteria such as subject and testing and assessment expertise, experience and quality are key in determining such appointments. Art education makes frequent use both of teaching staff without any teaching qualifications, and of external examiners. The appointment of examiners is one of the examination board’s discretionary powers, and one that even supersedes the authorisations of the executive board regarding personnel policy. This MORE INFORMATION means that the examination board’s recom- ArtEZ: Regulations mendations to the institution also impact the faculty’s recruitment & selection and profes- ArtEZ: Policy sional development policies. AHK: Assessment Appointment examinors Format exam CM Further Improvement Workshop report a coser look 47 Workshops ‘We must find language that we can use to express what it is we do’ Interview with Ton Lamers (ArtEZ), Marcel Doorduin (ArtEZ), Anton Neggers (Fontys Universities of Applied Sciences) and Jan Wirken (Zuyd University of Applied Sciences)

What was it like to meet and talk like this? Foreword Ton Lamers: ‘We reached agreement quite I Opening: Monique Vogelzang quickly, the topics we discussed were very familiar to everyone. We all felt we were II Interview: VSNU, VH, NRTO colleagues!’ III Message from the minister IV Interview: ISO Anton Neggers: ‘We had an animated discussion in which everybody shared V Workshops their opinions. We are all coming from the VI Interview: Hobéon, QANU, AeQui same place, and realise that cooperation is necessary. Art institutions often think they are VII Infomarket unique, but talking to each other reveals that VIII Presentation: Jan Anthonie Bruijn we all face the same challenges.’ IX Panel discussion Marcel Doorduin: ‘This session has helped us X Closing address: Monique Vogelzang realise that we’re all beating the same drum. Literature and links There are more similarities than differences.’ What was your most important message? Neggers: ‘One participant made a very important observation: that art educators V.15+16 must leave their egos at home during assessments. Instead of giving your opinion as an individual, you need to cooperate and 10+ operate within the context of the institution.’ Doorduin: ‘I believe it’s very important for us all to collaborate and learn from each other, and to find language that we can use to express what it is we do.’

Jan Wirken: ‘It should be viewed as an initial step in the collaboration among examination boards in the arts, which is MORE INFORMATION designed to encourage mutual inspiration and transparency.’ ArtEZ: Regulations ArtEZ: Policy AHK: Assessment Appointment examinors Format exam CM Further Improvement Workshop report a coser look 48 Workshops

Further Improvement a coser look 49 Workshops

‘We ask students to write a final project proposal, and formulate Foreword their own criteria outlining what I Opening: Monique Vogelzang it is they want to achieve. In addi- II Interview: VSNU, VH, NRTO tion to the general requirements, III Message from the minister these personal criteria are also IV Interview: ISO taken into account during assess- V Workshops ment. After all, students them- VI Interview: Hobéon, QANU, AeQui selves should also state the level VII Infomarket they want to attain.’ VIII Presentation: Jan Anthonie Bruijn Robert von der Nahmer IX Panel discussion Chairman of the Rotterdam Acad- X Closing address: Monique Vogelzang emy of Architecture and Urban Literature and links Design examination board

V ‘External experts may not assess final examinations for more than two years. This is to retain the external experts’ fresh perspective, 10+ and to prevent affiliations from becoming too strong.’

Cornelia Steenmeijer Member of the Minerva Academy central examination board, Hanze University of Applied Sciences, Groningen

Further Improvement a coser look 50 Workshops Toetsen, niet minder maar anders

by the handicap + studie Expertise Centre RELEVANCE TO ‘FURTHER IMPROVEMENT’ Lex Jansen (handicap + studie Ex- in which Casper, a student with Asperger In its report, the Inspectorate looks at examina- pertise Centre) syndrome, explains his difficulty with inter- tion boards’ responsibility to issue decisions on Foreword preting multiple-choice questions. individual students. The task of adapting tests for The studies of around 10% of all students I Opening: Monique Vogelzang students with functional disabilities falls under in higher education are affected by a func- The Equal Treatment of Disabled and II Interview: VSNU, VH, NRTO this responsibility, but is often very time-consum- tional disability or chronic illness such as Chronically Ill People Act (WGBH/CZ) states ing. During this workshop, the handicap + studie III Message from the minister dyslexia, ADD, ADHD, a disorder on the that students with functional disabilities or Expertise Centre presented a reference model to autism spectrum, or Crohn’s disease. Stat- chronic illnesses are entitled to adaptations. IV Interview: ISO assist in this process. utory provisions exist that entitle these Adapted conditions are already common- V Workshops students to adapted assessment, and it place: students may sometimes sit ‘sepa- is the examination board’s job to make rately’, or are given more time to complete VI Interview: Hobéon, QANU, AeQui sure it is provided. Such adaptations can a test. Adaptations to form, however, are VII Infomarket involve both the test format and the con- still sporadic, for fear of putting the reliabil- VIII Presentation: Jan Anthonie Bruijn ditions under which it is completed. Lex ity or validity of the test at risk. Such fears Jansen showed how forms of testing can are unfounded, said Jansen: examination IX Panel discussion be adapted without compromising reli- boards can use the expertise centre’s refer- X Closing address: Monique Vogelzang ability or validity. ence model to verify whether an alternative test is equivalent, or whether quality has Literature and links Many students with a functional disabili- been sacrificed. The model also offers sug- ty or chronic illness experience difficulty gestions on the admission and implementa- with the testing prescribed by study pro- tion processes for alternative testing within grammes. This is not because they lack the education institutions. required knowledge or skills: rather, it is V.17 due to the conditions of, or the form taken by, the test itself. To illustrate the problem, 10+ Jansen showed a short film commissioned ‘Adaptations to the form of a test are sporadic, for fear of MORE INFORMATION Assessment Reference Model Video: Casper putting its reliability at risk. Article: Own Assessment Poem Article: Deafness Workshop Presentation This fear is unfounded’ Article: Dyslexia Workshop handout Further Improvement a coser look 51 Workshops

Further Improvement a coser look 52 Workshops Fraud: from practice to policy

Ludo van Meeuwen (Eindhoven workshop concluded that there are three RELEVANCE TO ‘FURTHER IMPROVEMENT’ University of Technology) and main causes: study or other pressures, the The report states that all examination boards Foreword presence of an opportunity, and rationalisa- impose sanctions in instances of fraud. Anti-fraud Arie de Wild (Rotterdam Univer- tion. Because study pressure is due mostly measures are often quite limited, however, giving I Opening: Monique Vogelzang sity of Applied Sciences) to external factors and therefore difficult to rise to the risk that fraud may not always be dis- II Interview: VSNU, VH, NRTO impact via fraud policy, the workshop con- covered (or may be discovered too late). During this It is the job of examination boards to en- centrated on the latter two reasons. So what workshop, two education institutions demonstrat- III Message from the minister sure the quality of testing, which entails a can examination boards do to limit rationali- ed what they do to remain in control in this respect, IV Interview: ISO shared responsibility for the prevention of sation and the opportunities for fraud? and the role played by the examination board. V Workshops fraud. But how can instances of fraud be detected, and how can fraud be prevented Comprehensive fraud policy VI Interview: Hobéon, QANU, AeQui in the first place? Testing and assessment Van Meeuwen distinguished four import- VII Infomarket adviser Ludo van Meeuwen from Eind- ant steps in the structuring of comprehen- hoven University of Technology and Pro- sive fraud policy: information, prevention, VIII Presentation: Jan Anthonie Bruijn fessor of Behavioural Economics Arie de detection and sanctions (see box). Exam- However, this does not mean that no fraud IX Panel discussion Wild from Rotterdam University of Applied ination boards will work optimally within is taking place. For this reason, said Arie de Sciences offered some footholds for a their capacity by effecting all four anti-fraud X Closing address: Monique Vogelzang Wild, it is important for examination boards comprehensive anti-fraud policy. They also measures, said Van Meeuwen. If there is no to unearth the ‘hidden data’ pertaining to Literature and links presented a tool that examination boards initiative from elsewhere in the hierarchy, fraud. But how? De Wild outlined a pragmat- can use to reveal hidden data pertaining to or if fraud prevention policy is outdated, ic approach. fraud. the examination board must act on its own initiative. As opposed to the executive board, the man- Both Meeuwen and De Wild recently in- agement and teaching staff, students are V.18 vestigated the subject of fraud. Eindhoven One aspect of fraud policy to keep in mind often already aware of the hidden facts re- University of Technology has summarised its is that lecturers themselves can also contrib- garding fraud. Informal discussions among fraud policy in a new document outlining a ute to fraud. There are cases, for example, students can help to reveal this information. comprehensive approach, and in 2014 Rot- of lecturers who reveal the exact content of Circumvent the hierarchy, create a trusted 10+ terdam University of Applied Sciences used a an examination in advance, who neglect to atmosphere and avoid immediate docu- new research method to reveal hidden fraud monitor plagiarism, or who issue grades for mentation – chatting and taking minutes data. The aim of both institutions is to re- work not completed. There are also stories afterwards works better. If the atmosphere main in control with regard to integrity and of sleeping invigilators in circulation. Po- is right, you can then start to ask questions awarding diplomas. tential solutions to these problems include spot-checks by examination boards at exam- to try to get to the bottom of the situation. Don’t enquire about fraud directly, but ask Serious fraud: what is it, and why do peo- inations, an extra invigilator whose job it is general questions such as: ‘What are your ple commit it? to monitor the invigilators, and digital exam concerns about our institution?’ or ‘What First of all, it is important to create a clear submission to enable instant plagiarism would you be ashamed of if it ever came to MORE INFORMATION definition of what serious fraud actually checks. light?’ If there are instances of fraud within is. Some examples mentioned during the an institution, some students will report cas- Anti-fraud tool* workshop include intentional fraud, identity Hidden data es voluntarily. In this context, it is important fraud, full plagiarism, forging research data, The Further Improvement* report points out to remain sincere in your dealings with peo- Fraud policy fraud during inspection of exam papers, and the peculiarity of the fact that sometimes ple. The method requires that you constantly repeated fraud. The next important question only very few cases of fraud, if any, are detect- Further Improvement* maintain a critical view of permissible ways was: why do students commit fraud? The ed among large numbers of students. to use the data collected. >>

Further Improvement a coser look 53 *English translation Workshops

>> Anti-fraud tool Examination boards must act according to a Foreword realistic view of fraud practices within their institution. This anti-fraud tool can help THE FOUR STEPS TOWARDS COMPREHENSIVE FRAUD POLICY I Opening: Monique Vogelzang examination boards to gain insight into these II Interview: VSNU, VH, NRTO practices, and to help fulfil their anti-fraud 1. Inform students properly about what is permitted – this will make it more responsibilities. difficult for them to justify fraud afterwards. This step primarily targets the ‘rational- III Message from the minister isation’ motivator. Advise students (particularly at research universities) that fraud IV Interview: ISO is unacceptable in their studies: future scholars, after all, are expected to treat V Workshops confidential information in a reliable fashion.

VI Interview: Hobéon, QANU, AeQui 2. Prevent fraud. For example, if two students are known for copying each other’s VII Infomarket work, they should be separated. Make fraud more difficult by increasing the likelihood of exposure. This step reduces the opportunities for fraud. VIII Presentation: Jan Anthonie Bruijn IX Panel discussion 3. Detect fraud. For example, ensure professional supervision during examinations, so that it is clear precisely which students are cheating. The role of invigi- X Closing address: Monique Vogelzang lators is important in this respect. Literature and links 4. Impose sanctions on the culprits, making sure to keep closely to the applicable procedures. Aim for uniform sanctions. 10+V.18 ‘The main causes of fraud are study pressures, the presence of an opportunity, and rationalisation’ MORE INFORMATION Anti-fraud tool* Fraud policy Further Improvement*

Further Improvement a coser look 54 *English translation Workshops The examination board, the management and the executive board: A ‘dynamic’ relationship

Jan Wiss (Inholland) RELEVANCE TO ‘FURTHER IMPROVEMENT’ In its report, the Inspectorate advises all parties to Foreword Examination boards must be able to fulfil Inadequate facilitative support jointly decide on a clear and transparent position their role from a position of indepen- A second grievance, and one that exam- for examination boards within the context of their I Opening: Monique Vogelzang dence. In practice, they encounter a num- ination boards have been airing for years, institutions. In practice, the relationship between II Interview: VSNU, VH, NRTO ber of problems that can be attributed in is the lack of adequate facilitative support the examination board and the executive board/ for their members. Both the minister and management can be a complicating factor. This III Message from the minister part to their relationships with the executive board and the management. Legal the Inspectorate confirm this once again workshop looked at the tensions and problems IV Interview: ISO Affairs Manager Jan Wiss from Inholland in Further Improvement*, pointing to the that can arise in relation to the examination V Workshops University of Applied Sciences outlined institutions as the responsible parties. The board’s position within the institution. the problems, and also went in search institutions have other priorities, however, VI Interview: Hobéon, QANU, AeQui of possible solutions with members of meaning that the issue of support remains VII Infomarket examination boards. unresolved. As a consequence, the members – whose tasks are often combined with de- VIII Presentation: Jan Anthonie Bruijn The Higher Education and Research Act manding teaching duties – frequently leave IX Panel discussion (WHW) prescribes a position of indepen- their work for the examination board by the wayside, further increasing the risks. X Closing address: Monique Vogelzang dence for examination boards, from which to guarantee the value of a higher-educa- Literature and links tion diploma. At the same time, the boards The workshop participants suggested ex- themselves are also part of the organisation plicit documentation of support standards and maintain relationships with the man- for institutions (hours allocated for the agement and the executive board. These chairman, secretary and examination board V.19 relationships impose other responsibilities members), who must then incorporate that are sometimes at odds with the board’s those standards into support regulations. independent position. During the work- In addition, they suggested that each insti- shop several examination board members tution should have several employees ded- confirmed that these relationships are not icated entirely to working in and for exam- 10+ always ideal, and that the various interests ination boards. Based on these roles, these can create tensions. The important thing is people could also serve as external mem- for managers and examination boards not bers of other examination boards within to lose sight of the greater good – the value the institution. Support should also be of the diploma – even when relationships adequate for these purposes, and is perhaps are strained. a suitable issue for the political agenda.

MORE INFORMATION BFL Annual report template BFL Examination Boards BFL Assessment Committees Further Improvement* Further Improvement a coser look 55 *English translation Workshops Dilemmas in the work of examination board members

Frans de Vijlder (HAN) RELEVANCE TO ‘FURTHER IMPROVEMENT’ In its report, the Inspectorate advises all parties Foreword Examination boards bear a major re- to jointly decide on a clear and transparent posi- sponsibility, Since the introduction of tion for examination boards within the context of I Opening: Monique Vogelzang the Improved Governance (Higher Educa- their institutions. This sometimes proves difficult II Interview: VSNU, VH, NRTO tion) Act, the job of examination boards in practice, however. This workshop looked at the dilemmas faced by examination boards. III Message from the minister to ensure the quality of interim and final examinations has been much more ex- IV Interview: ISO plicit than before. How can they fulfil this

V Workshops responsibility, and what is their relationship with the executive board and faculty VI Interview: Hobéon, QANU, AeQui management? What problems do they VII Infomarket encounter in practice? Professor of Good Governance and Innovation Dynamics VIII Presentation: Jan Anthonie Bruijn Frans de Vijlder summarised these dilem- IX Panel discussion mas. X Closing address: Monique Vogelzang The objective of the Improved Governance Literature and links (Higher Education) Act is to enable higher education examination boards to properly fulfil their responsibilities using what is called ‘professionals governance’. The V.20 formation of networks and exchange of knowledge are key aspects in this approach. The search for the right way for examination boards to fulfil their role is a cyclical learning process, said De Vijlder. Knowledge 10+ circulation is crucial in this respect.

MORE INFORMATION Essay: Frans de Vijlder Article: Frans de Vijlder

Further Improvement a coser look 56 Workshops

‘Although networking is probably my main goal today, I’d also like Foreword to hear the latest news and set pri- I Opening: Monique Vogelzang orities for our examination board. II Interview: VSNU, VH, NRTO What is the current national stan- III Message from the minister dard? What are the issues? I want IV Interview: ISO to see how our board is doing. To me, the most important thing is V Workshops to find out how my fellow board VI Interview: Hobéon, QANU, AeQui members from all universities or- VII Infomarket ganise their guidelines for struc- VIII Presentation: Jan Anthonie Bruijn turing, entering and finalising re- IX Panel discussion sults. That is one of our problem X Closing address: Monique Vogelzang areas. We are currently looking at Literature and links available good practices, and at how we should move ahead. An- other important topic is the role of the examination board within V the organisation, in particular with regard to the executive board – where is the divide between 10+ their and our responsibilities?’ Elissaveta Radulova Member of the Maastricht University examination board

Further Improvement a coser look 57 Workshops Research universities and universities of applied sciences: learn from each other!

Susan Voogd (VU)

Foreword In the Further Improvement* report, stant feedback, and also documenting this examination boards express their willing- feedback in order to enable joint develop- I Opening: Monique Vogelzang ness to learn and share good practices. ment of testing and assessment expertise. II Interview: VSNU, VH, NRTO The sharing of knowledge presupposes All universities can learn from each other in this way. The workshop’s key conclusion III Message from the minister that the various parties are all familiar with the subject at hand, and that the was that adequate testing and assessment IV Interview: ISO concepts are clearly defined. In her work- expertise is required in both the manage- V Workshops shop, Susan Voogd pointed out details ment hierarchy and the examination board. to the participants that must be clarified Voogd added: ‘Be inspired by others. Test VI Interview: Hobéon, QANU, AeQui or taken into account within their own in- their thoughts on the matter.’ VII Infomarket stitutions. Voogd is a testing and assessment expert, and works at VU University VIII Presentation: Jan Anthonie Bruijn Amsterdam as chair of the examination RELEVANCE TO ‘FURTHER IMPROVEMENT’ IX Panel discussion board for higher-education healthcare In its report, the Inspectorate advises institutions programmes. to exchange knowledge. This workshop demon- X Closing address: Monique Vogelzang strated good practices from the conclusion of the Literature and links To illustrate the difficulties that arise due to report, and participants were called on to share vague definitions, Voogd started with the their knowledge. word ‘examiner’. The Higher Education and Research Act and the report titled ‘Responsi- V.21 ble Assessment and Decisions in Higher Ed- ucation’ (Verantwoord toetsen en beslissen in het hoger onderwijs) do not use the same definition. This means that faculty policy must clearly define exactly what ‘examiners’ 10+ are, as the definition used has consequenc- es for both examiners and others, as well as for the design of the test system. The issue raised various questions. For example, is it permissible to appoint multiple examiners for a single test? Is there a clear definition of ‘taking a test’? What criteria do we apply to the appointment of examiners, and can they be external parties? MORE INFORMATION Voogd emphasised the fact that examina- Good practices tion boards are already performing very Responsible testing well: they are testing and improving together, both within and outside of the institu- Further Improvement* tion. To safeguard quality, Voogd advised collaborative testing, giving each other con Workshop notes Further Improvement a coser look 58 *English translation Workshops

‘During the workshop, it was im- ‘I picked up a lot of good ideas portant to me to hear how other during the workshop. What I Foreword institutions were doing. It’s good found most inspiring were the I Opening: Monique Vogelzang to be able to verify your own prac- stories about short lines of com- II Interview: VSNU, VH, NRTO tices against those of others. It munication at some universities III Message from the minister was also interesting to hear about of applied sciences. The presen- IV Interview: ISO the similarities and differences tation was very clear, and it was V Workshops between research universities and interesting to hear about the dif- VI Interview: Hobéon, QANU, AeQui universities of applied sciences. ferences between research univer- I thought they would be worlds sities and universities of applied VII Infomarket apart, but they were not as dissim- sciences. We have a large student VIII Presentation: Jan Anthonie Bruijn ilar as I expected, which is useful population at the University of IX Panel discussion to know.’ Twente, and international stu- X Closing address: Monique Vogelzang dents are a major focus. The visi- Literature and links Elly Holthuizen bility of test quality is particularly University Medical Center Utrecht, important for this group.’ Utrecht University V.21 Hèla Klaczynski Secretary, University of Twente 10+ examination board

MORE INFORMATION Good practices Responsible testing Further Improvement* Workshop notes Further Improvement a coser look 59 *English translation Workshops Testing and assessment, and the limited evaluation of study programmes

Sietze Looijenga (QANU), Hester Minnema (Leiden University) and Ivo van Stokkum Foreword (VU University Amsterdam)

I Opening: Monique Vogelzang Inspection panels always meet with the accreditation framework (which incorpo- II Interview: VSNU, VH, NRTO examination board during their inspec- rated both the testing system and the level tions. What approach do they take? What achieved). Faculties that received an unsatis- III Message from the minister are their experiences with examination factory evaluation have proven quite able to IV Interview: ISO boards? And how can inspection panels respond effectively, said Looijenga. He cited V Workshops help examination boards to perform examples from faculties that had taken the better? These are the questions that Si- critical comments by inspection panels to VI Interview: Hobéon, QANU, AeQui etze Looijenga, Director of Quality Assur- heart in order to improve their testing and VII Infomarket ance Netherlands Universities (QANU), assessment system, giving quality a substan- discussed during this workshop. QANU tial boost. The Faculty of Mathematics and VIII Presentation: Jan Anthonie Bruijn performs inspections at research univer- Natural Sciences at the University of Gronin- IX Panel discussion sities. Hester Minnema and Ivo van Stok- gen, for example, has drastically altered its kum presented one of their university’s testing and assessment system in response X Closing address: Monique Vogelzang good practices. to the evaluation of the university’s chemis- RELEVANCE TO ‘FURTHER IMPROVEMENT’ Literature and links try programmes, which have now made the In its report, the Inspectorate notes that Tijdens een visitatie richten panels zich grade. decisions by the Accreditation Organisation of Looijenga began by saying that during an the Netherlands and Flanders (NVAO) and the inspection, a panel will concentrate more The Bachelor of Criminology at Leiden underlying observations from inspection reports on the results of an examination board’s University is another example. Hester correspond to the results of the inquiry. Nearly V.22 activities, rather than on the question of Minnema, education manager at the Leiden all faculties that are currently undergoing an whether the board adequately fulfils all of University Faculty of Law, related her experi- improvement process report problems with the its statutory duties. Only if a panel is criti- ences. The inspection panel concluded that examination board. This workshop looked at the cal of the quality of testing or final assign- the Bachelor of Criminology final thesis was view taken during inspections of testing in general 10+ ments/submissions will it scrutinise the sub-par. Although the examination board and examination boards in particular, and showed method behind the assessment. This will in- was operating effectively, it was still in the that inspection panels can have a major impact on clude the formal procedures and guidelines process of finding ways to fulfil its statutory the performance of examination boards. established by the examination board and, responsibilities. The panel advised creating as a logical extension thereof, the board’s a schedule for the phased implementation own performance. This way, inspection of its intensified role, and the faculty devot- panels that carry out a limited performance ed explicit attention to this recommenda- assessment can exert a major influence on tion during its improvement activities. For the quality and safeguarding of testing and example, further professional development MORE INFORMATION assessment. was provided for examination board members (through training and selection), as Interview with Looijenga, et al. University of Groningen and Leiden Uni- well as intensification of quality assurance versity: major improvements via the faculty examination board. And the Presentation: Looijenga Nearly all ‘unsatisfactory’ grades issued un- results? The NVAO awarded the programme Presentation: Minnema til now by the inspection panels related to a final grade of ‘good’ instead of ‘unsatisfac- the third standard in the NVAO’s previous tory’ for standard 3. >> Presentation: Van Stokkum Further Improvement a coser look 60 Workshops

>> VU University Amsterdam examination- tions is safeguarded: every course has a second the next resit to the assessment committee, board stands out lecturer who shares responsibility for the tests; which then ran correctly. The assessment During its inspection, said Looijenga, the Fac- final assignments are assessed by two inde- committee also checked the examination the Foreword ulty of Sciences at VU University Amsterdam pendent staff members; and plagiarism checks following year, which contained a completely stood out straight away in a positive sense. are mandatory for all tests. The assessment new set of questions. I Opening: Monique Vogelzang This was due in part to the quality assurance of committee (a sub-committee of the examina- II Interview: VSNU, VH, NRTO its testing and assessment, in which the faction board) will investigate any examinations Majority of test systems effective III Message from the minister ulty examination board played an important for which exceptional results were obtained, Looijenga concluded the workshop by observ- part. Ivo van Stokkum, lecturer and member or that were problematic the year before. ing that most academic/research programmes IV Interview: ISO of multiple boards including this examination use effective test systems, and that there are V Workshops board, explained their practices. To illustrate, Von Stokkum discussed a case in examples of programmes that exceed the basic which the assessment committee successful- level. Faculties that do not reach the standard VI Interview: Hobéon, QANU, AeQui First of all Van Stokkum described the advan- ly intervened after a remarkable number of at the time of their inspection are usually able VII Infomarket tages of a single faculty examination board as 9s and 10s had been awarded for a particular to take appropriate measures quickly. opposed to many separate boards. These ad- examination. It turned out that all of the mul- VIII Presentation: Jan Anthonie Bruijn vantages include increased flexibility, oppor- tiple-choice questions had been recycled. The IX Panel discussion tunities to make use of each other’s expertise, results of the examination were declared inval- X Closing address: Monique Vogelzang and the practical benefit of having to produce id, and all students were given the opportunity only one annual report. He went on to explain to resit at their convenience. For validation Literature and links how the quality of interim and final examina- purposes, the examiner was ordered to submit V.22 10+ ‘Following improvement activities, the Criminology programme at UL was awarded

MORE INFORMATION a final grade of “good” instead Interview with Looijenga, et al. Presentation: Looijenga of “unsatisfactory” Presentation: Minnema Presentation: Van Stokkum Further Improvement a coser look 61 Workshops Ensuring quality and standards in joint higher education programmes with international partner institutions

Peter Wieringa and Marinke Taken together, these developments present Sussenbach (Delft University of the various institutional stakeholders with Foreword new challenges. How can the quality of such Technology) initiatives be safeguarded? Wieringa and I Opening: Monique Vogelzang Sussenbach demonstrated Delft University II Interview: VSNU, VH, NRTO Education institutions are offering more of Technology’s experiences in the field of RELEVANCE TO ‘FURTHER IMPROVEMENT’ and more forms of online teaching, online and joint education programmes. The report outlines how the plans to make higher III Message from the minister and are increasingly operating in an The university has set up a special test education more flexible will increase the need IV Interview: ISO international context. One example framework that can assist institutions for further improvement among examination of this international focus is joint boards. Developments such as exchange and joint V Workshops in the creation of fully-fledged joint education programmes, which are education programmes. The test framework education programmes also challenge examina- VI Interview: Hobéon, QANU, AeQui offered in conjunction with interna- is available upon request from Marinke tion boards to find new ways of safeguarding tional institutions. Vice-rector of quality. Delft University of Technology outlined VII Infomarket Sussenbach: [email protected]. Delft University of Technology Peter these challenges during the workshop. VIII Presentation: Jan Anthonie Bruijn Wieringa and Delft Extension School IX Panel discussion and Joint Education policy officer Marinke Sussenbach looked at how these X Closing address: Monique Vogelzang developments affect examination boards. Literature and links Forms of collaboration between Dutch institutions and international partners are expected to multiply in the years ahead, possibly resulting in complications for ‘Dutch institutions must V.23 examination boards. For example: when providing joint international or other tracks/programmes, Dutch institutions comply with both Dutch must comply with both Dutch legislation 10+ and the quality requirements of the partner institution abroad, which are not always compatible. In addition, legislation and the examination boards have had to deal with stricter legislation in recent years, while the number of requirements has not quality requirements of diminished. the partner institution abroad’

Further Improvement a coser look 62 Workshops

‘I am interested to see how admis- ‘I’m the sidekick during one of sions procedures at universities the workshops, but that’s not the Foreword of applied sciences will develop, main reason I’m here. Thankfully I Opening: Monique Vogelzang as they can be very complex. Take I’m here to present the workshop, II Interview: VSNU, VH, NRTO students from China, for example. because the event was booked out III Message from the minister What are their diplomas worth? quite soon. I really wanted to be IV Interview: ISO Can they be admitted? Are they here. There is so much happening V Workshops entitled to exemptions to certain right now that affects examina- subjects? How do we deal with tion boards, and this is the ideal VI Interview: Hobéon, QANU, AeQui issues of this kind? I have plen- opportunity to stay up-to-date VII Infomarket ty of experience in this area, and and hear about the approaches VIII Presentation: Jan Anthonie Bruijn am eager to share what I know. I taken by other people.’ IX Panel discussion also hope to hear other people’s X Closing address: Monique Vogelzang perspectives on certain issues. Hester Minnema Literature and links So today’s event is quite import- Education Manager, Leiden Uni- ant; all of the relevant actors and versity stakeholders are right here.’

V Peter Nonhof Admissions, Stenden University of 10+ Applied Sciences

Further Improvement a coser look 63 Interview: ‘The rewards of investments Hobéon, QANU, AeQui are evident’

Interview with Wienke Blomen primary role as that of an “administrative” (Hobéon), Sietze Looijenga body – not an uncommon view ten years Foreword ago – have effectively become a thing of the (QANU) and René Kloosterman past. Examination boards are increasingly I Opening: Monique Vogelzang (AeQui) adopting a more subject-oriented perspective II Interview: VSNU, VH, NRTO in line with the amendments to the Act. III Message from the minister Assessment agencies evaluate higher-ed- They are creating are clearer identity for Wienke Blomen ucation study programmes, and their themselves, feel more responsible for Director Hobéon Groep BV IV Interview: ISO inspection panels speak to a wide variety everything related to testing and assessment, V Workshops of examination boards. What have they and anticipate more. Board members are found out over the past few years? What generally more prepared for the work they VI Interview: Hobéon, QANU, AeQui problems do they typically encounter, and do, have often attended training courses or VII Infomarket what advice do they give? Directors Wienke workshops, and are aware of the statutory Blomen (Hobéon), Sietze Looijenga requirements and the duties and responsi- VIII Presentation: Jan Anthonie Bruijn (QANU) and René Kloosterman (AeQui) bilities they entail.’ IX Panel discussion share their insights below. X Closing address: Monique Vogelzang Wienke Blomen: ‘The reports from our How have examination boards developed panels have shown that examination boards Literature and links over the last few years? have improved significantly in recent years. René Kloosterman: ‘Examination boards There have been major investments in more are far more active than they used to be. and better forms of facilitative support With improvements to both professional for examination boards, as well as in their VI development and their identity within identity and professional development. institutions, they are now all busy making The rewards are evident, as shown by the progress. What is evident is that the rate of results of the Inspectorate’s inquiry. Still, the René Kloosterman progress relies heavily on the board members reports by our panels also highlight areas Director AeQui themselves. In general, examination boards for improvement, such as the identity of 10+ examinations boards, their relationship with are progressing at a reasonable rate; some are lagging behind somewhat, and others examiners (who are also their colleagues) and progress so quickly that they risk “losing the support they receive (see box titled ‘Areas touch” with education.’ for improvement’).’

Sietze Looijenga: ‘In recent years, much Could you give examples of typical has changed in research-oriented education recommendations given by the inspection with regard to examination boards’ panels to examination boards? performance and the interpretation of their Kloosterman: ‘The recommendations that duties. To a large extent, these changes are we give are always related to the perceived the result of amendments to the Higher rate of progress that I mentioned before. So Education and Research Act, and of an we either recommend becoming a little more increased focus on quality assurance within pro-active, or to keep the integration with institutions in response to the implemen- study programmes in mind. One recommen- tation of the institutional quality assurance dation that we never give during inspections test. Examination boards who see their but that I would like to mention all the >> Sietze Looijenga Director QANU

Further Improvement a coser look 64

>> same, is to try to make yourselves redundant contributed ideas to developments in the field. Interview: within three years. Examination boards are, and They involved a number of people in a feedback have always been, a safeguarding instrument; group, and consulted with 200 examination the actual quality of tests is realised elsewhere. boards in round-table discussions during the Hobéon, QANU, Devoting attention to the greater whole will inquiry. The report had already won support ultimately result in better education.’ among higher education institutions before the AeQui ink had dried. The same success is evident today: Looijenga: ‘The most common recommen- rather than resisting the Inspectorate, people – dation that we have always issued is to be more the boards especially – feel partly supported by pro-active. Panels believe that examination the results. This has also set the tone for today. Foreword boards should monitor the quality of testing The inquiry has been quite pleasant.’ actively and improve it where necessary. I Opening: Monique Vogelzang Another common recommendation is for René Kloosterman: ‘This is the Inspectorate, II Interview: VSNU, VH, NRTO examination boards to promote consistency and version 2.0!’ III Message from the minister transparency in testing and assessment. This advice often pertains to the assessment of final IV Interview: ISO assignments or theses, e.g. if panels believe that SCALING UP V Workshops the assessment forms used are either not good enough or are completed incorrectly.’ Sietze Looijenga notes a striking development in how examination boards are organised: upscaling. VI Interview: Hobéon, QANU, AeQui What are the benefits? VII Infomarket Blomen: ‘One good example comes from a university of applied sciences that I visited ‘In the past, nearly every study programme had its own examination board. But now, more and VIII Presentation: Jan Anthonie Bruijn recently, which used an incredibly detailed more examinations board are responsible for multiple programmes, such as all programmes within IX Panel discussion assessment form for final theses. In fact it a faculty, school or graduate school. This increase in scale offers clear advantages in terms of professional development, specialisation, efficiency and the workloads of teaching staff. The most obvious X Closing address: Monique Vogelzang was so comprehensive that the assessors felt overwhelmed, and were ultimately unable to drawback is a drop in subject-related expertise. Examination boards can compensate for this quite Literature and links demonstrate whether students had actually effectively, however, by setting up sub-committees that contain representatives from specific attained the competencies or not. Of course, study programmes. One of the presentations during my workshop was about this type of broader, the recommendations issued by our panels faculty-wide examination board. depend on the specific context, and may relate to the board’s statutory tasks, prioritisation, and This development is part of an important cultural shift currently taking place among universities, VI operationalisation of quality requirements (see one effect of which is a decrease in the traditional autonomy of academic staff. Another is the fact ‘Recommendations for Examination Boards’).’ that they can also be held accountable for the teaching they provide and the tests they administer, including the results.’ What do you think of today’s event? Kloosterman: ‘It’s fantastic that they organised it. The fact that they issued such an interesting report and then followed through with a day like this is great.’

Looijenga: ‘I agree completely. Just perfect.’

Blomen: ‘The Inspectorate did sometimes demonstrate typical “inspector” behaviour in the past: conducting inquiries in secret, failing to incorporate the opinions of the institutions under inspection in their reports, and – sometimes rightly so – setting a negative tone. That is not the case here. They have been very open about the inquiry from the start, and have

Further Improvement a coser look 65

Interview: TAKING ACTION IMPROVEMENT AREAS Although many examination boards are making Wienke Blomen describes four problem areas regularly observed by the panels, and the opportu- Hobéon, QANU, improvements, there are faculties where nities for development they represent. progress is lagging. Sietze Looijenga described AeQui the factors that play a role. Struggling with identity ‘The subject of identity applies not only to examination boards, but also to management and the ‘It is often the smaller programmes that get left executive board, and mostly concerns the operationalisation and optimisation of the examination behind. For example, since 2010 humanities board’s independence. Everyone involved – but executive boards, management, examiners and Foreword programmes have devoted a lot of time lecturers in particular – still does not always see the full scope and implications of this process as to curriculum modifications as part of the self-evident. It is a developmental process, true, and there is nothing wrong with that. It is also I Opening: Monique Vogelzang Sustainable Humanities programme, and in about the examination board’s authority. Examination boards must acquire this authority through II Interview: VSNU, VH, NRTO preparation for the 2012 and 2013 inspections. professional, expert action and appropriate communication, but it also depends on adminis- Examination boards were put on the back burner trative and managerial commitment. The executive board and management must support the III Message from the minister during this time, which probably partly explains examination board, and their commitment must be beyond all doubt. This, too, is a develop- IV Interview: ISO why the examination boards of a relatively large mental process.’ number of humanities programmes received V Workshops critical feedback from inspection panels. Delicate relationship with fellow examiners VI Interview: Hobéon, QANU, AeQui ‘Many examiners are also colleagues of examination board members. The smaller the faculty, VII Infomarket These boards are usually those responsible the more the delicate nature of these relationships becomes obvious. Examination boards for a single programme or for a combination appoint examiners (and can revoke these appointments), and issue guidelines and instructions. VIII Presentation: Jan Anthonie Bruijn of a Bachelor’s programme and subsequent Study programmes need time to adjust, especially those that have not used the appointment of IX Panel discussion Master’s, that is, smaller boards that have not examiners as a quality assurance instrument in the past. But again: this process takes time.’ been absorbed into larger ones. These boards X Closing address: Monique Vogelzang feel that it is harder to do justice to the subject- Scope of interim and final examinations (and quality assurance) Literature and links specific aspects of testing as part of a larger ‘Some examination boards are still unclear on the scope of the concepts of “interim examination” examination board. They also tend to have and “final examination”. Examination boards must safeguard the quality of all aspects of testing problems with the fact that the new role of and assessment: an area that relates to all work produced or behaviours demonstrated by examination boards infringes on the traditional students that are assessed as part of the study programme. This assessment will then determine autonomy of academic staff, who teach and whether course credits are awarded or not. The scope therefore extends beyond traditional VI administer tests without others getting involved examinations and final assignments/theses. Examination boards are becoming increasingly or knowing exactly what it is they do. In these aware of this fact. Attention to learning and testing “on the job” is still an area for improvement situations, the necessary cultural shift is delayed, within many faculties and examination boards. or takes longer. There is also a growing recognition of the fact that quality assurance involves more than just I believe that the examination boards with a supervision. It is about taking pro-active steps: setting quality requirements for the design and more entrenched traditional approach will implementation of testing and assessment and for examiners, through the guidelines and be forced to “modernise” their practices and instructions the board issues. It must also act responsively, by monitoring implementation procedures in the near future. Otherwise, through spot-checks, the intensity of which may decrease as confidence in the examiners grows.’ executive boards and deans consider the risks to study programmes (e.g. during inspections) to be Differences in support due to underestimated workload too great.’ ‘Support in terms of time, budget, administrative and subject-related support, training, etc. must be determined by the scope and nature of an examination board’s duties and the extent to which they are outsourced. In practice, we have noticed that institutions are still trying to strike a balance – the hallmark of a change process that has not yet fully crystallised.’

Further Improvement a coser look 66

Interview: ADVICE FOR EXAMINATION BOARDS Five practical recommendations issued regularly by inspection panels are Hobéon, QANU, summarised below by Wienke Blomen.

AeQui Focus on your statutory duties ‘Generally, examination boards that are still busy making the transition from “bookkeeper” to “watchful eye” spend most of their energy on ensuring the effective fulfilment of their statutory duties. In such cases, panels recommend Foreword leaving aside any additional tasks, such as admissions or the binding negative study advice.’ I Opening: Monique Vogelzang II Interview: VSNU, VH, NRTO Set priorities ‘Rome wasn’t built in a day. Examination boards must set priorities based on III Message from the minister importance, urgency and capacity, to be set out in a development schedule IV Interview: ISO covering a period of around two years. To examination boards still at the ‘The report outset of their development, panels often recommend focusing initially on V Workshops targeted research into the quality of testing and assessment during the first VI Interview: Hobéon, QANU, AeQui and final years of the study programme.’ had already VII Infomarket Make quality criteria operational VIII Presentation: Jan Anthonie Bruijn ‘It is useful for examination boards to work in conjunction with the faculty won support IX Panel discussion to translate criteria such as reliability, validity, transparency, objectivity/ intersubjectivity, recognisability and feasibility into concrete requirements and X Closing address: Monique Vogelzang mechanisms.’ among higher Literature and links Organise and direct calibration sessions ‘However objective an assessment framework may be, its interpretation and education application can vary from examiner to examiner. It is important, therefore, VI for the relevant examiners to arrive at an optimum consensus regarding the key considerations within each section of an assessment framework that institutions determine a grade – unsatisfactory, satisfactory, or good.’ Clarify the types of support that are necessary before the ink ‘Discuss the types of support that are essential to successful performance explicitly with the executive board and management, and insist on fixed agreements.’ had dried’

Further Improvement a coser look 67 Infomarket

Further Improvement a coser look 68 WorkshopsInfomarket A stroll through the Infomarket Exhibitors take the floor

Voorwoord During the conference, informa- ‘Two years ago we put together a

I Opening Monique Vogelzang tion stands had been set up by booklet titled “Testing: Not less, Foreword “Als vereniging werken we hard aan onze II Interview VSNU, various people and institutions just different”, in order to aid ex- naamsbekendheid, want ons werk is I Opening: Monique Vogelzang Vereniging Hogescholen, with something special to report amination boards when asked for precies voor deze mensen bedoeld. Onze II Interview: VSNU, VH, NRTO on the performance of examina- altered forms of testing for stu- doelstellingen zijn het behartigen van NRTO de belangen voor examencommissies, III Message from the minister tion boards. Once of the stands dents with a functional disability. III Boodschap van de minister toetsmakers en beleidsmakers. Om het IV Interview: ISO was devoted to good practices: We know that this idea still makes eindniveau te waarborgen onderzoeken IV Interview ISO V Workshops practices that institutions were people nervous, because how can we samen wat examens tot kwalitatief V Workshops keen to share due to their own you know whether the person goede examens maakt. Wanneer deze dag VI Interview: Hobéon, QANU, AeQui geslaagd is voor ons? Nu al: we staan hier, VI Interview Hobéon, QANU, AeQui positive experiences. The Higher has attained the required com- VII Infomarket hebben een aantal interessante gesprekken VII Informatiemarkt Education Press Agency (Hoger petencies? This booklet enables gehad en tot nu toe komen we alleen maar VIII Presentation: Jan Anthonie Bruijn VIII Presentatie Jan Anthonie Bruijn Onderwijs Persbureau, HOP) modifications to be made without enthousiasme tegen. We zijn gesterkt in IX Panel discussion het feit dat onze vereniging bestaansrecht IX Paneldiscussie also attended, and wrote an ar- compromising on quality. I think heeft.” X Closing address: Monique Vogelzang X Slotwoord Monique Vogelzang ticle on the conference. Some it’s a pity that the Inspectorate’s Barbara Suijkerbuijk Literature and links exhibitors share their experienc- report doesn’t mention it explicit- Bestuurslid Nederlandse Vereniging voor Achtergronden binnen handbereik es below. ly. Today I hope to speak to lots of Examens (NVE)

people, and underline its impor- “Twee jaar geleden maakten wij het boekje tance.’ Toetsen, niet minder maar anders. Dat is VIIV.- een hulpmiddel voor examencommissies, als er gevraagd wordt om een andere Lex Jansen toetsvorm voor mensen met een beperking. Senior consultant, handicap + Wij weten dat daar nogal huiverig mee 10+ studie Expertise Centre omgegaan wordt, want hoe weet je dan of iemand aan de gestelde competenties voldoet? Met dit boekje kun je zonder kwaliteitsverlies verandering aanbrengen. Ik vind het jammer dat het niet expliciet in het rapport van de inspectie staat. Vandaag hoop ik met meer mensen te kunnen praten en het belang te onderstrepen.” Lex Jansen MORE INFORMATION Senior consultant Expertisecentrum handicap + studie Overview of stands “Ik vond het vooral heel leerzaam en HOP Article belangrijk om vandaag een kijkje in de Good practices keuken te nemen. Ik ben zelf voorstander van een flexibeler onderwijs. Naar mijn idee

Further Improvement a coser look 69 WorkshopsInfomarket

Voorwoord ‘We are working hard on the visi- ‘I found it particularly instructive I Opening Monique Vogelzang bility of our association, because and important to come and take Foreword II Interview VSNU, our work is targeted precisely at a behind-the-scenes look today. I Opening: Monique Vogelzang Vereniging Hogescholen, the people here today. Our objec- Personally, I am in favour of more II Interview: VSNU, VH, NRTO tive is to represent the interests of flexible education. I think that NRTO III Message from the minister examination boards, test writers more variation in testing is a good III Boodschap van de minister IV Interview: ISO and policymakers. To safeguard thing, and today I spoke to many IV Interview ISO V Workshops exit levels, we conduct joint re- people who share this opinion. I V Workshops VI Interview: Hobéon, QANU, AeQui search into what it takes to create believe that examination boards VI Interview Hobéon, QANU, AeQui good-quality exams. When can we can play an important part in VII Infomarket VII Informatiemarkt consider today a success, you ask? this.’ VIII Presentation: Jan Anthonie Bruijn VIII Presentatie Jan Anthonie Bruijn It’s already a success: we’re here, IX Panel discussion IX Paneldiscussie have already had a number of Irma van Slooten X Closing address: Monique Vogelzang X Slotwoord Monique Vogelzang interesting discussions and until Coach, Universal Design for Literature and links now we have met with nothing Learning (UDL) Netherlands Achtergronden binnen handbereik but enthusiasm. We take heart in the fact that our association clear- VIIV.- ly has a raison d’être.’ Barbara Suijkerbuijk Executive Board Member, Dutch 10+ Association for Examinations (NVE)

MORE INFORMATION Overview of stands HOP Article Good practices

Further Improvement a coser look 70 WorkshopsInfomarket

Voorwoord ‘I am very interested in the ‘The Good Practices stand I Opening Monique Vogelzang experiences of examination attracted a lot of visitors. People Foreword II Interview VSNU, boards in higher professional rummaged through the folders, I Opening: Monique Vogelzang Vereniging Hogescholen, and senior secondary vocational and even copied down tips and II Interview: VSNU, VH, NRTO NRTO education. We currently operate examples on the spot. We are III Message from the minister in private education, and I glad that our knowledge is being III Boodschap van de minister IV Interview: ISO am keen to find out how they shared this way. Incidentally, the IV Interview ISO V Workshops maintain independence in the good practices are available to V Workshops VI Interview: Hobéon, QANU, AeQui creation and assessment of everybody via this e-magazine.’ VI Interview Hobéon, QANU, AeQui VII Infomarket interim and final examinations. VII Informatiemarkt There is new legislation (good Anne-Marie van Rijsbergen VIII Presentation: Jan Anthonie Bruijn VIII Presentatie Jan Anthonie Bruijn legislation, if you ask me), and Higher Education Inspector, IX Panel discussion IX Paneldiscussie I am curious as to how these Inspectorate of Education X Closing address: Monique Vogelzang X Slotwoord Monique Vogelzang schools organise it and what Literature and links Achtergronden binnen handbereik problems they encounter. Today we’re here to help. We’ve been doing this for 70 years, and are VIIV.- happy to share what we know. We even have our own stand, so offering assistance will be no 10+ problem at all.’ Astrid Hanou Quality Assurance Officer, Dutch Association for Examinations

MORE INFORMATION Overview of stands HOP Article Good practices

Further Improvement a coser look 71 Presentation: Working on education’s Jan Anthonie Bruijn weakest link

Jan Anthonie Bruijn

Foreword Jan Anthonie Bruijn is well-known from 2008 the Dijsselbloem report was issued, the External Validation Report (Vreemde which looked at the question of whether the I Opening: Monique Vogelzang ogen dwingen), but is also a member of education system could be improved. The II Interview: VSNU, VH, NRTO the Dutch Senate, professor of immuno- essence of the report was: let’s separate the pathology at Leiden University, ex-mem- ‘what’ from the ‘how’, and test the ‘wheth- III Message from the minister ber of the Education Council and member er’ more effectively (i.e. whether the learn- IV Interview: ISO of the Utrecht University Supervisory ing objectives are being achieved). That is what the examination boards are all about. V Workshops Board. His work therefore covers various levels of education and examination. In VI Interview: Hobéon, QANU, AeQui his plenary presentation, he outlined Setting an agenda: the ins and outs VII Infomarket the frameworks for a practical agenda for Today is a special day, because we are examination boards. working on the weakest link in our already VIII Presentation: Jan Anthonie Bruijn strong education system. I did not realise IX Panel discussion Today I would like to speak to you with only until at least ten years after I had created my one of my many hats on: that of lecturer and first university examinations that the exam- X Closing address: Monique Vogelzang examiner in research-oriented education. ination board even existed, let alone that Literature and links When I arrived this morning, I was struck they had appointed me as an examiner. by the unique location, in which trains take centre stage. Examination boards are also The role and visibility of examination on board a moving train. Attendance today boards has changed a lot in recent years. is twice as high as anticipated, which is But what can be improved? According to VIII great to see. Perhaps the name of the In- the Inspectorate’s report, the board’s duties spectorate of Education should be changed and identity could be defined much more to ‘Inspiration of Education’! It’s wonderful clearly, and by that I mean the relationships that the Inspectorate has organised this between the various committees, clear 10+ event, and that so many parties from higher communication and concise prioritisation. education are here today. Topics such as teaching methods are the domain of the education committee, an The Dutch education system is one of the area that the examination board should best in the world, due to the quality of our simply stay away from. Likewise, we still see teachers and lecturers, and to society’s too many programme committees elbowing involvement in education. My personal in on testing and examination. Commit- experience as a lecturer has taught me that tees often communicate with other bodies testing and assessment is the weakest link which – according to the organisational MORE INFORMATION in our system. Perhaps that is why it is so chart – they have nothing to do with. Of exciting for all of us to be here today. If we course this can sometimes be useful, but all External Validation Report can do something about it, we will be im- too often it is counterproductive. So keep it proving all of education: the most import- simple, and aim for ‘deregulation’ instead of Responsible testing ant portfolio there is. creating more regulations. Incidentally, no- where does the Further Improvement report Dijsselbloem Committee So we are all on board a moving train. In suggest even one additional regulation. >> Further Improvement* Presentation Further Improvement a coser look 72 *English translation >> Breaking free of mediocrity professional skills up to date. I like this system, Presentation: Nowadays all examination boards must have at especially when viewed from a patient’s per- least one external member, in order to guaran- spective. My proposal is that a similar system tee the independence of quality assurance. An- be introduced into education: set minimum Jan Anthonie other way to foster independence is to engage requirements for the testing and assessment external examiners. One organic way to do this expertise of examiners, set a standard for pro- Bruijn is through joint examinations. This is possible fessional development, and most of all allow when using ‘snapshot’ exams, however in 1977 the professional community to formulate that Maastricht University introduced a much more standard themselves through an ongoing pro- innovative instrument: the learning path- cess and discussion. Foreword way-independent, institution-wide progress test. This test is administered to all students, In 2013, via an expert group, we launched the I Opening: Monique Vogelzang say, every three months, and assesses them at BKE and SKE examiner accreditation standards II Interview: VSNU, VH, NRTO the programme’s exit level. in the hope that they would trigger internal dis- III Message from the minister cussions among education institutions. Many The Maastricht progress test has several advan- institutions did take up the idea and some IV Interview: ISO tages: it facilitates stronger institutional iden- agencies included a certification system, so that V Workshops tity in accordance with Veerman, and is trans- now there are plenty of avenues for profession- parent. The test also offers a major economic al development. Quality still remains in the VI Interview: Hobéon, QANU, AeQui advantage, as it involves much less work than hands of the institutions themselves, but from VII Infomarket unit-based testing, and no resits are required. an external quality assurance perspective, it’s a Also, students’ typical study tactics (quickly good thing they are running with it. VIII Presentation: Jan Anthonie Bruijn reviewing the old questions the day before) do IX Panel discussion not work, and a more constant investment of The question is: should training also be com- X Closing address: Monique Vogelzang energy is rewarded. If we were to implement pulsory for examiners? Will we take the same this as a standard method, we would break free path as the medical profession? One possible Literature and links of our ‘culture of mediocrity’ straight away, argument against this idea is the fact that no because it brings added value and the benefits retraining is required for people with a driver’s of study into focus. license, for example. But that’s not to say that road safety wouldn’t improve if we all did half Examiners’ registry a day’s training every five years, to brush up on VIII As a doctor I need to be registered, so that pa- new legislation and road signals. tients can see whether I satisfy a certain min- >> 10+ imum standard when it comes to keeping my ‘As a teacher it took me at least ten years

MORE INFORMATION before I realised that External Validation Report Responsible testing the examination board Dijsselbloem Committee Further Improvement* even existed’ Presentation Further Improvement a coser look 73 *English translation >> Towards a practical agenda Presentation: As far as I could tell, I was never officially appointed by an examination board. I never knew about it. Now I do, but I have never Jan Anthonie spoken to any members of the examination board. In fact, the Inspectorate’s inquiry re- Bruijn vealed that one third of all examiners are unaware of their own appointment. In turn, one third of examination boards state that they would not know what to do in the event that Foreword examiners produce poor-quality exams. Many improvements can be made in this regard. I Opening: Monique Vogelzang II Interview: VSNU, VH, NRTO We must work towards an agenda that has III Message from the minister more practical applicability. Are you all clear on what your duties are, and – more impor- IV Interview: ISO tantly – what they are not? Look at your re- V Workshops sponsibilities, and see whether you can do anything about your independence. And what VI Interview: Hobéon, QANU, AeQui about your expertise? We talk about BKE, but VII Infomarket actually it’s about setting the bar for yourself. Find out whether your exam papers cover VIII Presentation: Jan Anthonie Bruijn all competencies, and agree on a maximum IX Panel discussion allowable percentage of recycled questions. X Closing address: Monique Vogelzang Have outsiders come and observe – yes, research universities, even you – and watch your Literature and links tests improve as part of quality agreements. Prize efficiency in all that you do, as our workload is already too great. We must reduce the amount of regulation, limit ourselves to our core duties and create smart, innovative in- VII struments that can improve both quality and efficiency, such as the Maastricht progress test. 10+ Lastly: know thyself, and use the tools available to you. We have an incredibly effective education system. Trust people, but remember to verify their work as well.

MORE INFORMATION External Validation Report Responsible testing Dijsselbloem Committee Further Improvement* Presentation Further Improvement a coser look 74 *English translation Panel- discussion

Further Improvement a coser look 75 Panel- Students on the examination board? discussion Plenty of insights for further improvement

The workshops, interviews and presen- the examination boards in the various who asked her fellow panel members and tations all offered plenty of food institutions is what struck me in particular. the audience to respond to the following: Foreword for thought. So what should we do Of course diversity is not an issue, provided ‘The value of a diploma cannot be now? What can be done with all the that every examination board is pursuing its guaranteed without a properly functioning I Opening: Monique Vogelzang information gathered today? The own developmental course. I am very glad examination board.’ II Interview: VSNU, VH, NRTO plenary panel discussion offered some therefore that there are 600 people here initial suggestions. Mediated by Cor today; we can learn a lot from each other. Students are the primary focus of education, III Message from the minister Ottens, panel members Liesbeth Zijlstra But diversity should not be the result of Zijlstra argues, and students must have IV Interview: ISO (Amsterdam University of Applied some examination boards and managers the opportunity to obtain a degree that is V Workshops Sciences, LOEx), Nellie Harms (VU choosing to settle for lower standards.’ worth something. A large majority of the University Amsterdam), Martine Pol audience stood up. Still, some criticism VI Interview: Hobéon, QANU, AeQui (Inspectorate of Education) and Lex A valuable diploma was expressed. One audience member said VII Infomarket Sietses (former Dutch National Union Some of the propositions also involved the that the statement’s formulation was too of Students Executive Board member) audience: Ottens asked them to stand if narrow: ‘There are other factors besides VIII Presentation: Jan Anthonie Bruijn shared their opinions, and put forward they were in agreement with the statement the examination board that determine the IX Panel discussion some propositions to the audience. under discussion. The microphone was quality of education.’ handed to Liesbeth Zijlstra, chair of the X Closing address: Monique Vogelzang First of all, Ottens asked Further Amsterdam University of Applied Sciences Students on the examination board Literature and links Improvement project manager Martine examination board and of the National Lex Sietses was involved in the inquiry as a Pol what surprised her most during her Examinations Board Consultative Body student, and is in favour of the proposition work. Pol: ‘The level of diversity among (Landelijk Overleg Examencommissies), that examination boards should >> 10+IX

Further Improvement a coser look 76 >> also contain student members: ‘No Panel- students from the same faculty, but from other faculties. Students’ main criticism of examination boards is that they are not discussion transparent. Students wonder about the reasoning behind certain decisions, and believe the board to be invisible.’ According to Sietses, students can help bring about change by joining examination boards themselves, bridging the gap between the Foreword boards and the student body and increasing the support base. I Opening: Monique Vogelzang II Interview: VSNU, VH, NRTO Many attendees were hesitant about giving III Message from the minister students a place on the examination board. Nellie Harms, education director at the IV Interview: ISO Faculty of Earth and Life Sciences at VU V Workshops University Amsterdam, wondered what risks such a decision would entail. She herself VI Interview: Hobéon, QANU, AeQui supports the proposition that examination VII Infomarket boards have the tendency to try to take over the job of the programme director. Zijlstra VIII Presentation: Jan Anthonie Bruijn and Sietses said they would rather turn the IX Panel discussion proposition around: in their experience, X Closing address: Monique Vogelzang it is usually the managers who try to usurp the role of the examination board. The Literature and links audience was equally divided on the issue.

Rewards No consensus was reached during the discussion, but that was not the point. IX Everybody could apply the insights gained today in their own way, to help further improve their own examination board. 10+ That was the true reward of the day’s event.

Further Improvement a coser look 77 Panel- discussion

Further Improvement a coser look 78 Foreword I Opening: Monique Vogelzang II Interview: VSNU, VH, NRTO III Message from the minister IV Interview: ISO V Workshops VI Interview: Hobéon, QANU, AeQui VII Infomarket VIII Presentation: Jan Anthonie Bruijn IX Panel discussion X Closing address: Monique Vogelzang 10+ Literature and links

Further Improvement a coser look 79 Panel Follow-up discussie Minister Jet Bussemaker asked all attendees to give thought to how they plan to follow up on today’s proceedings. What will you take away with you? Read three responses below. Voorwoord I Opening Monique Vogelzang ‘I already have a few “quick wins” ‘First thing tomorrow I’ll be II Interview VSNU, VH, NRTO in mind for tomorrow. First of all sending the examiners their III VerenigingBoodschap Hogescholen, van de minister we’ll discuss the guidelines we letters of appointment – an IV NRTOInterview ISO plan to issue to our examiners. involved but important job. Second, we’ll create a registry of Maybe we’ll digitise it for next VIII WorkshopsBoodschap van de minister our own appointed examiners, year. One question struck me VIIV Interview Hobéon,ISO QANU, AeQui which we will then publish in today: should students be allowed VIIV WorkshopsInformatiemarkt order to ensure that students and to sit on the examination board? VIIIVI PresentatieInterview Hobéon, Jan Anthonie QANU, Bruijn AeQui other examiners know them. In Our organisation will certainly IXVII InformatiemarktPaneldiscussie the long term, we wish to take be discussing that one. I’ll be XVIII SlotwoordPresentatie Monique Jan Anthonie Vogelzang Bruijn part in the flexibility pilot. We interested to see what extra tasks IX LiteratuurPaneldiscussie en links need to think about a potential will become part of our remit. X Slotwoord Monique Vogelzang new role for our examination All of the topics discussed today Achtergronden binnen handbereik board.’ will also be part of tomorrow’s meeting among the examination Ad van Zundert board chairs. We plan to power VIII Chair of the Part-Time Academy ahead.’ examination board, Avans University of Applied Sciences Charlotte Faber Official secretary, University of 10+ Amsterdam Graduate School for ‘On the one hand, I can see that Humanities we are not doing too badly at all. On the other, international examination quality requires attention and needs to be more tightly controlled.’ MORE INFORMATION Conference evaluation Esther Brinkhoff Examination Board member, Avans University of Applied Sciences

Further Improvement a coser look 80 Closing Closing address: Monique address Vogelzang

Foreword I Opening: Monique Vogelzang ‘I believe that today was a success. II Interview: VSNU, VH, NRTO I’ve met many enthusiastic and III Message from the minister inspiring people. Our aim was IV Interview: ISO to give the report back to the professionals so that they can V Workshops get to work, and that is what we VI Interview: Hobéon, QANU, AeQui achieved. We also wanted to draw VII Infomarket attention to the changes that VIII Presentation: Jan Anthonie Bruijn need to occur, and it is wonderful IX Panel discussion that they were highlighted today. X Closing address: Monique Vogelzang I would like to thank everybody Literature and links for their attendance today. The discussion certainly doesn’t end here, there is still plenty to do. The glass is half full, and I hope to IX see you again in the future, with even fuller glasses.’ 10+ Monique Vogelzang Inspector-General, Inspectorate of Education

Further Improvement a coser look 81 Literature Literature and links and links

Many articles in this magazine disability.] Examens [Examinations], 1, 9-13. Inspectorate of Education (2009). contain links to background Higher Education Appeals Tribunal (CBHO, Boekhouder of wakend oog? Verslag van Foreword 2015). De examencommissie in de jurispru- een onderzoek bij examencommissies in materials, either as part of the text dentie van het College van Beroep voor het hoger onderwijs over de garantie van I Opening: Monique Vogelzang or under ‘More information’. het Hoger Onderwijs. [The examination het niveau. [Bookkeeper or watchful eye? II Interview: VSNU, VH, NRTO board in the legal precedents of the Higher Report on an inquiry into quality assurance Education Appeals Tribunal.] The Hague: among higher education examination III Message from the minister These can include PowerPoint Higher Education Appeals Tribunal. boards.] Utrecht: The Inspectorate of IV Interview: ISO presentations, the workshop Education. V Workshops handouts or good practices: Committee for the external validation of examples of procedures that examinations quality in higher education Inspectorate of Education (2015a). VI Interview: Hobéon, QANU, AeQui (2012). Vreemde ogen dwingen. Eindrapport Verdere versterking. Onderzoek naar het institutions believe worked VII Infomarket Commissie externe validering examenk- functioneren van examencommissies in het well within their own context waliteit hoger beroepsonderwijs. [External hoger onderwijs. [Further Improvement: VIII Presentation: Jan Anthonie Bruijn and that they would like to Validation: Final report by the committee Inquiry into the performance of higher IX Panel discussion for the external validation of examination education examination boards.] Utrecht: share with their fellow organi- quality in higher education]. The Hague: The Inspectorate of Education. X Closing address: Monique Vogelzang sations. Links are also included Netherlands Association of Universities of Literature and links to documents beyond the scope Applied Sciences (HBO-raad). Inspectorate of Education (2015b). Good of the workshops, such as related practices. Hoofdstuk uit rapport Verdere handicap + studie Expertise Centre versterking. [Good practices: chapter from articles and reports. (Expertisecentrum handicap + studie, 2013). the Further Improvement report.] Utrecht: Toetsen, niet minder maar anders. Referen- The Inspectorate of Education. The list below presents the tiemodel.[Testing: not less, just different. VIII A reference model.] ‘s Hertogenbosch: Inspectorate of Education (2015c). Relevante more general sources; click on handicap + studie Expertise Centre. wetsartikelen WHW. Bijlage uit rapport the relevant title to go to that Verdere versterking. [Relevant articles from 10+ publication. The literature list Hooge, Edith (2014). Hoge verwachtingen, the Higher Education and Research Act is followed by good practices, vrije uitvoering, stevige sturing. Een essay (WHW): Annex to the Further Improvement over onderwijsbestuur. [High expectations, report.] Utrecht: The Inspectorate of divided into six sections. free implementation, strong guidance: An Education. essay on education governance.] The Hague: Advisory board for flexible adult higher Ministry of Education, Culture and Science Dutch National Student Association (ISO, education (2014). Flexibel hoger onderwijs (OCW). 2014). De student centraal. Knelpunten- voor volwassenen. Adviesrapport. [Flexible notitie over de rechtspositie van studenten. Adult Higher Education: Advisory Report] Huisman, P.W.A., & Vijlder, F. de (2012). De [Focus on students: bottleneck report on The Hague: Ministry of Education, Culture sleutelrol van professionals governance. students’ legal status.] Utrecht: Dutch and Science (OCW). Over de reputatieverschillen tussen hbo National Student Association (Interstedelijk en wo. [The key role of professionals Studenten Overleg, ISO). Boxtel, A. van, Jansen, J., Jansen, L., & governance: On the reputation differences Burg, J. ter (2014). Ieder zijn eigen toets? between universities of applied sciences and UDL als oplossing voor studeren met een research universities.] Thema, 3, 26-31. functiebeperking. [Individual testing? UDL as a solution for studying with a functional

Further Improvement a coser look 82 Jaspers, M., & Zijl, E. van (2014). sonderwijs. Een voorstel voor een Literature Samenwerken aan toetskwaliteit in het programma van eisen voor een basis- en hoger onderwijs. [Working together on seniorkwalificatie examinering (BKE/ the quality of testing in higher education.] SKE). [Responsible testing and decisions in and links Eindhoven: Fontys Universities of Applied higher professional education: A proposed Sciences. schedule of requirements for basic and senior examination qualifications (BKE/ Knecht-van Eekelen, A. de (2014). Terecht of SKE).] Report by the BKE/SKE expert group. niet? Eens dyslectisch – altijd dyslectisch. The Hague: Netherlands Association of [Fair or not? Once dyslexic, always dyslexic.] Universities of Applied Sciences. Foreword Examens [Examinations], 2, 14-15. Netherlands Association of Universities I Opening: Monique Vogelzang Lamers, T. (2014). Gepubliceerd beleid van of Applied Sciences (2014). Beoordelen II Interview: VSNU, VH, NRTO de examencommissie. [Policy published is mensenwerk. Bevindingen over de III Message from the minister by examination boards.] Examens wenselijkheid en mogelijkheid van een [Examinations], 2, 11-13. gezamenlijk protocol voor het beoordelen IV Interview: ISO van (kern)werkstukken. [Assessment is not V Workshops Lei, R. van der (2014). Examinatoren a formula: Observations on the desirability beoordeeld. Externe validering door het of and opportunities for a joint protocol VI Interview: Hobéon, QANU, AeQui BKE-traject van de Hanzehogeschool for the assessment of core (and other) VII Infomarket Groningen. [Examiners assessed: External assignments.] Report by the Protocol expert validation by the BKE assessment accredi- group. The Hague: Netherlands Association VIII Presentation: Jan Anthonie Bruijn tation programme at the Hanze University of Universities of Applied Sciences. IX Panel discussion of Applied Sciences, Groningen]. Examens X Closing address: Monique Vogelzang [Examinations], 4. Netherlands Association of Universities of Applied Sciences (2015). Handreiking Literature and links Dutch National Union of Students (LSVb, examencommissies 2015. [Examination 2014). Onderzoek naar de rol van examen- Committee Guide 2015.] The Hague: commissies in het hoger onderwijs. [Study Netherlands Association of Universities of on the role of examination boards in higher Applied Sciences. education.] Utrecht: Dutch National Union of Students (LSVb). Vijlder, F. de (2015). Professionals aan VIII het roer. Goed bestuur door profes- Ministry of Education, Culture and Science sionals. [Professionals at the helm: Good (OCW, 2014). Leven lang leren. [Lifelong governance by professionals.] In: Veld, R. 10+ learning.] [Letter] to the Lower House of in ‘t (ed.). De echte dingen. Essays over de the Dutch Parliament, 31 October 2014. The kwaliteit van onderwijs [The right stuff: Hague: Ministry of Education, Culture and Essays on the quality of education] (126-138). Science. Amsterdam: Gopher B.V.

(OCW, 2015). Rapport ‘Verdere versterking’ over het functioneren examencommissies. [Further Improvement report on the performance of examination boards.] [Letter] to the Lower House of the Dutch Parliament, 20 April 2015. The Hague: Ministry of Education, Culture and Science.

Netherlands Association of Universities of Applied Sciences (2013). Verantwoord toetsen en beslissen in het hoger beroep-

Further Improvement a coser look 83 Literature Links good practices and links

CHAPTER 1.a Regulations of examination boards Foreword I Opening: Monique Vogelzang II Interview: VSNU, VH, NRTO CHAPTER 1.b Regulations of examination boards III Message from the minister IV Interview: ISO V Workshops CHAPTER 2.a Reports VI Interview: Hobéon, QANU, AeQui VII Infomarket VIII Presentation: Jan Anthonie Bruijn CHAPTER 2.b Reports IX Panel discussion X Closing address: Monique Vogelzang

CHAPTER 2.c | Literature and links Reports

CHAPTER 3 Profiles of examination boards and examiners

CHAPTER 4.a 10+ Testing and assessment

CHAPTER 4.b Testing and assessment

CHAPTER 4.c Testing and assessment

CHAPTER 5 Fraud

CHAPTER 6 Further information

Further Improvement a coser look 84 Foreword I Opening: Monique Vogelzang II Interview: VSNU, VH, NRTO III Message from the minister IV Interview: ISO V Workshops VI Interview: Hobéon, QANU, AeQui VII Infomarket VIII Presentation: Jan Anthonie Bruijn IX Panel discussion X Closing address: Monique Vogelzang 10+ Literature and links

Further Improvement a coser look 85 EXAMINATION BOARDS in higher education

Issued by the Inspectorate of Education Coordinator: Martine Pol Editor: Antje Visser Design: Visuele Notulen Translation: Metamorfose Vertalingen June 2015 Further Improvement a coser look 86