Final Evaluation of the Francommunautés Virtuelles Program FINAL REPORT February 2008 Tabled and approved at DEC on June 9, 2008 This publication is available upon request in accessible formats. Contact: Multimedia Services Section Communications and Marketing Branch Industry Canada Room 264D, West Tower 235 Queen Street Ottawa ON K1A 0H5 Tel.: 613-948-1554 Fax: 613-947-7155 Email: [email protected] Permission to Reproduce Except as otherwise specifically noted, the information in this publication may be reproduced, in part or in whole and by any means, without charge or further permission from Industry Canada, provided that due diligence is exercised in ensuring the accuracy of the information reproduced; that Industry Canada is identified as the source institution; and that the reproduction is not represented as an official version of the information reproduced, nor as having been made in affiliation with, or with the endorsement of, Industry Canada. For permission to reproduce the information in this publication for commercial redistribution, please email: [email protected] Cat. No. Iu4-137/2008E-PDF ISBN 978-1-100-10135-4 60485 Aussi offert en français sous le titre Évaluation finale du programme Francommunautés virtuelle. Translated version. In instances of inconsistency, the French version shall prevail. TABLE OF CONTENTS EXECUTIVE SUMMARY ................................................................................................. i 1.0 INTRODUCTION .................................................................................................. 1 2.0 BACKGROUND ................................................................................................... 2 2.1 History .................................................................................................................. 2 2.2 Program profile ..................................................................................................... 2 2.3 Program management and governance ............................................................... 4 2.3.1 Advisory Committee................................................................................... 4 2.3.2 Program team ............................................................................................ 4 2.3.3 National Selection Committee.................................................................... 5 2.4 Funding ................................................................................................................ 5 2.5 Recent evaluations ............................................................................................... 6 3.0 METHODOLOGY AND APPROACH................................................................... 7 3.1 Evaluation questions addressed........................................................................... 7 3.2 Evaluation methods .............................................................................................. 7 3.2.1 Roles and responsibilities .......................................................................... 7 3.2.2 Evaluation calendar and strategy............................................................... 7 3.2.3 Description of the lines of inquiry ............................................................... 7 3.3 Method limitations................................................................................................. 9 4.0 RESULTS........................................................................................................... 11 4.1 Relevance .......................................................................................................... 11 4.1.1 Program definition and direction .............................................................. 11 4.1.2 Better identification of winning needs ...................................................... 13 4.1.3 Who does Francommunautés virtuelles target?....................................... 14 4.2 Degree of success in achieving results .............................................................. 15 4.2.1 Project selection process......................................................................... 15 4.2.2 Program accessibility............................................................................... 16 4.2.3 Governance structure .............................................................................. 19 4.2.4 Competitive process management .......................................................... 20 4.3 Cost-effectiveness .............................................................................................. 23 4.3.1 CFA needs assessment and dissemination of information ........................ 23 4.3.2 Budgetary review....................................................................................... 24 4.3.3 Creation of networks and partnerships ...................................................... 24 5.0 CONCLUSION ................................................................................................... 26 APPENDIX 1 – Evaluation matrix APPENDIX 2 – Evaluation participants APPENDIX 3 – Case study analysis APPENDIX 4 – List of documents reviewed APPENDIX 5 – List of interviewees APPENDIX 6 – Interview Guides APPENDIX 7 – Survey questionnaire NOTE: Appendices are available via an Access to Information Request. List of Figures Figure 1 Distribution of Amounts Awarded, by Sector................................................... 12 Figure 2 Funding Distribution by Component and Competition ..................................... 17 Figure 3 FV Governance Structure ............................................................................... 19 List of Tables Table 1: FV Program Funding from 1998 to 2008 ........................................................... 5 Table 2: Operating and Maintenance Expenses and Salaries......................................... 5 Table 3: Number and Value of Contribution Agreements per Competition...................... 6 Table 4: Number of Individuals Interviewed .................................................................... 8 Table 5: Characteristics of the Test Population (Case Studies) ...................................... 8 Table 6: Project Selection Criteria................................................................................. 16 Table 7: Number of Contribution Agreements Granted to Provinces and Territories .... 18 Table 8: Answers from responding recipients that commented on directives, support, and the attainment of initial objectives.............................. 21 Table 9: Case Studies: Success factors, difficulties and local repercussions identified.......................................................................................................... 22 Table 10: Number of partnerships developed or built by category ................................ 25 Audit and Evaluation Branch Final Evaluation for the Francommunautés Virtuelles Program –Final Report February 2008 LIST OF ACRONYMS AC Francommunautés virtuelles program Advisory Committee AEB Audit and Evaluation Branch CBED Community-based economic development CFAs Francophone and Acadian communities1 CFDC Community Futures Development Corporation CIUS Canadian Internet Use Survey EDOLMC Economic Development of Official Language Minority Communities FCFA Fédération des communautés francophones et acadiennes du Canada FV Francommunautés virtuelles IC Industry Canada ICT Information and communications technology IHAB Information Highway Applications Branch NPO Non-profit organization OLMCs Official Language Minority Communities PCH Canadian Heritage RDÉE Réseau de développement économique et d'employabilité SME Small and medium-sized enterprise TBS Treasury Board Secretariat 1 These represent all Francophone and Acadian communities (CFAs), whether they are minorities or not. Consequently, Official Language Minority Communities (OLMCs) are included in the CFAs. Audit and Evaluation Branch Final Evaluation for the Francommunautés Virtuelles Program –Final Report February 2008 EXECUTIVE SUMMARY Context and objectives of the evaluation This final evaluation of the Francommunautés virtuelles program (FV) was carried out by Groupe-conseil baastel ltée on behalf of the Audit and Evaluation Branch (AEB) of Industry Canada (IC). Since 1998, this program has provided financial support for projects that aim to meet the specific needs of Francophone and Acadian communities (CFAs) in using information and communications technologies (ICTs). This evaluation takes into account the history of the program, while focusing on the activities undertaken between 2003 and 2007. It is intended to measure the success of the program. Methodology In order to carry out the evaluation, the evaluation team and the steering committee chose four lines of inquiry. They conducted an exhaustive documentation review, 31 interviews with key stakeholders, 9 case studies and an electronic survey of participants in the competitive process. Key findings The Francommunautés virtuelles program has had positive effects on communities. Nonetheless, there are several grey areas in the definition of FV, and this lack of clarity is reflected in its implementation on various levels. The program mission and objectives seem relevant, but are very broad. Some participants consider it essential that the funded projects help develop or maintain their competitiveness in a knowledge-based economy, whereas others emphasize the importance of bridging the digital divide. This dichotomy between innovation