D5.3 Evaluation of the final WeGov Toolbox 19 October 2012 SEVENTH FRAMEWORK PROGRAMME THEME ICT 2009.7.3 ICT for Governance and Policy Modelling Project acronym: WeGov Project full title: Where eGovernment meets the eSociety Grant agreement no.: 248512 D5.3 Evaluation of the final WeGov Toolbox Deliverable Id: D5.3 Deliverable Name: Evaluation of the final WeGov Toolbox Status: Final Dissemination Level: Public Due date of deliverable: 30 SePtember 2012 (Project month 33) Actual submission date: 19 October 2012 (Project month 34) Work Package: WP5 Scenarios, Testbeds and Evaluation Organisation name of lead contractor for this GESIS – Leibniz Institute for the Social Sciences deliverable: Author(s): Timo Wandhöfer, Catherine Van Eeckhaute, Beccy Allen, Steve Taylor, Paul Walland Partner(s) contributing: Gov2u, Hansard Society, IT Innovation Centre Abstract: This document is the evaluation report of the final WeGov toolbox. This document gives an overview of the WeGov prototype versions, describes the evaluation methodology with a process model and an evaluation model, and considers end users’ and expert advisers’ engagement. Complementary, in depth experiments were carried out to validate the accuracy and reliability of the toolbox 3.0. The WeGov methodology also considers the exemplary usage, a user guidance including a manual and Q&A, and privacy considerations. WeGov Consortium Page 1 of 217 D5.3 Evaluation of the final WeGov Toolbox 19 October 2012 History Version Date Modification reason Modified by 0.1 30.08.2012 Initial draft Timo Wandhöfer 0.2 30.09.2012 Improved version Timo Wandhöfer Catherine Van Eeckhaute 0.3 05.10.2012 WeGov Privacy Considerations Steve Taylor Paul Walland 0.4 06.10.2012 HeadsUp experiment Beccy Allen 0.5 08.10.2012 1st version for internal review Timo Wandhöfer Catherine Van Eeckhaute 0.6 09.10.2012 Modified structure Timo Wandhöfer Mark Thamm 0.7 11.10.2012 Review Beccy Allen 0.8 15.10.2012 2nd version for internal review Timo Wandhöfer Catherine Van Eeckhaute 0.9 17.10.2012 Review Steve Taylor Paul Walland Peter Mutschke 1.0 19.10.2012 Finale Version Timo Wandhöfer Catherine Van Eeckhaute WeGov Consortium Page 2 of 217 D5.3 Evaluation of the final WeGov Toolbox 19 October 2012 Table of contents HISTORY ............................................................................................................................... 2 TABLE OF CONTENTS ........................................................................................................ 3 LIST OF FIGURES ................................................................................................................ 7 LIST OF TABLES ................................................................................................................... 9 LIST OF ABBREVIATIONS .................................................................................................. 10 GLOSSARY .......................................................................................................................... 12 EXECUTIVE SUMMARY ...................................................................................................... 14 1 INTRODUCTION ................................................................................................................. 16 1.1 WEGOV WORK PACKAGE 5 DELIVERABLES ............................................................................... 18 1.2 HOW TO READ THE DOCUMENT .............................................................................................. 19 2 BACKGROUND .................................................................................................................. 20 2.1 METHODOLOGY ................................................................................................................... 20 2.1.1 PROCESS MODEL ....................................................................................................... 20 2.1.2 APPLIED PROCESS MODEL ......................................................................................... 22 2.1.3 EVALUATION MODEL ................................................................................................ 23 2.1.4 QUALITATIVE RESEARCH .......................................................................................... 24 2.1.5 APPLIED EVALUATION MODEL ................................................................................. 25 2.2 END USERS ......................................................................................................................... 27 2.2.1 EU-PARLIAMENT ....................................................................................................... 29 2.2.2 GERMAN PARLIAMENT .............................................................................................. 31 2.2.3 STATE PARLIAMENT .................................................................................................. 31 2.2.4 LOCAL GOVERNMENT ............................................................................................... 31 2.2.5 CITY ........................................................................................................................... 32 2.2.6 PARLIAMENTARY PARTY ........................................................................................... 32 2.2.7 NGO .......................................................................................................................... 32 2.3 EXPERT ADVICE .................................................................................................................... 33 2.3.1 ADVISORY BOARD ..................................................................................................... 33 2.3.2 WORKSHOPS .............................................................................................................. 34 WeGov Consortium Page 3 of 217 D5.3 Evaluation of the final WeGov Toolbox 19 October 2012 2.3.3 EVENTS ...................................................................................................................... 34 2.3.4 EU COMMISSION ........................................................................................................ 34 2.4 CHRONOLOGICAL OVERVIEW OF STAKEHOLDER ENGAGEMENT CONTACTS ............. 35 3 EVALUATION OF THE PENULTIMATE TOOLBOX ................................................................. 38 3.1 INTRODUCTION .................................................................................................................... 38 3.2 TOOLBOX 2.5 ...................................................................................................................... 38 3.2.1 CHARACTERISTICS ..................................................................................................... 38 3.2.2 END USER FEEDBACK ................................................................................................ 42 3.2.3 RESULTS .................................................................................................................... 42 3.2.4 ADVISORY BOARD FEEDBACK .................................................................................. 48 3.3 TOOLBOX 2.6 ...................................................................................................................... 50 3.3.1 CHARACTERISTICS ..................................................................................................... 50 3.4 CONCLUSION OF THE EVALUATION OF THE PENULTIMATE TOOLBOX .............................................. 52 4 EVALUATION OF THE FINAL TOOLBOX ............................................................................... 54 4.1 TOOLBOX 3.0 ...................................................................................................................... 54 4.2 CHARACTERISTICS PRE VERSION .............................................................................................. 54 4.3 VALIDATION OF ANALYSIS RESULTS WITH END USERS ................................................................. 56 4.3.1 INTRODUCTION .......................................................................................................... 56 4.3.2 AIMS OF THE EVALUATION ........................................................................................ 56 4.3.3 METHODOLOGY AND SPECIFICATIONS ...................................................................... 57 4.3.4 RESULTS .................................................................................................................... 65 4.3.5 DISCUSSION ............................................................................................................... 80 4.3.6 CONCLUSION FOR VALIDATION OF ANALYSIS RESULTS WITH END USERS .............. 82 4.4 EVALUATION WITH THE EU-PARLIAMENT ................................................................................. 86 4.5 WORKSHOPS WITH SCIENTISTS, END USERS AND PRACTITIONERS ................................................. 89 4.5.1 INTRODUCTION .......................................................................................................... 89 4.5.2 2ND WEGOV WORKSHOP DURING EGOV12 CONFERENCE ...................................... 89 4.5.3 WORKSHOP WITH THE BUNDESTAG .......................................................................... 92 4.5.4 WORKSHOP WITH PRACTITIONERS FROM THE POLITCAMP12 ................................... 93 4.5.5 ADVISORY BOARD INVOLVEMENT ............................................................................ 95 WeGov Consortium Page 4 of 217 D5.3 Evaluation of the final