Final Evaluation of the Francommunautés Virtuelles Program
FINAL REPORT
February 2008
Tabled and approved at DEC on June 9, 2008
This publication is available upon request in accessible formats. Contact:
Multimedia Services Section Communications and Marketing Branch Industry Canada Room 264D, West Tower 235 Queen Street Ottawa ON K1A 0H5
Tel.: 613-948-1554 Fax: 613-947-7155 Email: [email protected]
Permission to Reproduce Except as otherwise specifically noted, the information in this publication may be reproduced, in part or in whole and by any means, without charge or further permission from Industry Canada, provided that due diligence is exercised in ensuring the accuracy of the information reproduced; that Industry Canada is identified as the source institution; and that the reproduction is not represented as an official version of the information reproduced, nor as having been made in affiliation with, or with the endorsement of, Industry Canada.
For permission to reproduce the information in this publication for commercial redistribution, please email: [email protected]
Cat. No. Iu4-137/2008E-PDF ISBN 978-1-100-10135-4 60485
Aussi offert en français sous le titre Évaluation finale du programme Francommunautés virtuelle.
Translated version. In instances of inconsistency, the French version shall prevail.
TABLE OF CONTENTS
EXECUTIVE SUMMARY ...... i 1.0 INTRODUCTION ...... 1 2.0 BACKGROUND ...... 2 2.1 History ...... 2 2.2 Program profile ...... 2 2.3 Program management and governance ...... 4 2.3.1 Advisory Committee...... 4 2.3.2 Program team ...... 4 2.3.3 National Selection Committee...... 5 2.4 Funding ...... 5 2.5 Recent evaluations ...... 6 3.0 METHODOLOGY AND APPROACH...... 7 3.1 Evaluation questions addressed...... 7 3.2 Evaluation methods ...... 7 3.2.1 Roles and responsibilities ...... 7 3.2.2 Evaluation calendar and strategy...... 7 3.2.3 Description of the lines of inquiry ...... 7 3.3 Method limitations...... 9 4.0 RESULTS...... 11 4.1 Relevance ...... 11 4.1.1 Program definition and direction ...... 11 4.1.2 Better identification of winning needs ...... 13 4.1.3 Who does Francommunautés virtuelles target?...... 14 4.2 Degree of success in achieving results ...... 15 4.2.1 Project selection process...... 15 4.2.2 Program accessibility...... 16 4.2.3 Governance structure ...... 19 4.2.4 Competitive process management ...... 20 4.3 Cost-effectiveness ...... 23 4.3.1 CFA needs assessment and dissemination of information ...... 23 4.3.2 Budgetary review...... 24 4.3.3 Creation of networks and partnerships ...... 24 5.0 CONCLUSION ...... 26
APPENDIX 1 – Evaluation matrix APPENDIX 2 – Evaluation participants APPENDIX 3 – Case study analysis APPENDIX 4 – List of documents reviewed APPENDIX 5 – List of interviewees APPENDIX 6 – Interview Guides APPENDIX 7 – Survey questionnaire
NOTE: Appendices are available via an Access to Information Request.
List of Figures Figure 1 Distribution of Amounts Awarded, by Sector...... 12 Figure 2 Funding Distribution by Component and Competition ...... 17 Figure 3 FV Governance Structure ...... 19
List of Tables Table 1: FV Program Funding from 1998 to 2008 ...... 5 Table 2: Operating and Maintenance Expenses and Salaries...... 5 Table 3: Number and Value of Contribution Agreements per Competition...... 6 Table 4: Number of Individuals Interviewed ...... 8 Table 5: Characteristics of the Test Population (Case Studies) ...... 8 Table 6: Project Selection Criteria...... 16 Table 7: Number of Contribution Agreements Granted to Provinces and Territories .... 18 Table 8: Answers from responding recipients that commented on directives, support, and the attainment of initial objectives...... 21 Table 9: Case Studies: Success factors, difficulties and local repercussions identified...... 22 Table 10: Number of partnerships developed or built by category ...... 25
Audit and Evaluation Branch Final Evaluation for the Francommunautés Virtuelles Program –Final Report February 2008
LIST OF ACRONYMS
AC Francommunautés virtuelles program Advisory Committee AEB Audit and Evaluation Branch CBED Community-based economic development CFAs Francophone and Acadian communities1 CFDC Community Futures Development Corporation CIUS Canadian Internet Use Survey EDOLMC Economic Development of Official Language Minority Communities FCFA Fédération des communautés francophones et acadiennes du Canada FV Francommunautés virtuelles IC Industry Canada ICT Information and communications technology IHAB Information Highway Applications Branch NPO Non-profit organization OLMCs Official Language Minority Communities PCH Canadian Heritage RDÉE Réseau de développement économique et d'employabilité SME Small and medium-sized enterprise TBS Treasury Board Secretariat
1 These represent all Francophone and Acadian communities (CFAs), whether they are minorities or not. Consequently, Official Language Minority Communities (OLMCs) are included in the CFAs.
Audit and Evaluation Branch Final Evaluation for the Francommunautés Virtuelles Program –Final Report February 2008
EXECUTIVE SUMMARY
Context and objectives of the evaluation
This final evaluation of the Francommunautés virtuelles program (FV) was carried out by Groupe-conseil baastel ltée on behalf of the Audit and Evaluation Branch (AEB) of Industry Canada (IC). Since 1998, this program has provided financial support for projects that aim to meet the specific needs of Francophone and Acadian communities (CFAs) in using information and communications technologies (ICTs). This evaluation takes into account the history of the program, while focusing on the activities undertaken between 2003 and 2007. It is intended to measure the success of the program.
Methodology
In order to carry out the evaluation, the evaluation team and the steering committee chose four lines of inquiry. They conducted an exhaustive documentation review, 31 interviews with key stakeholders, 9 case studies and an electronic survey of participants in the competitive process.
Key findings
The Francommunautés virtuelles program has had positive effects on communities. Nonetheless, there are several grey areas in the definition of FV, and this lack of clarity is reflected in its implementation on various levels.
The program mission and objectives seem relevant, but are very broad. Some participants consider it essential that the funded projects help develop or maintain their competitiveness in a knowledge-based economy, whereas others emphasize the importance of bridging the digital divide. This dichotomy between innovation and the digital divide leads to confusion as to the target population.
In addition, in 2003, IC renewed the FV program by integrating it into its Action Plan for Official Languages. However, the program targets not only Official Language Minority Communities (OLMCs), but also the entire Francophone population in Canada, an inconsistency pointed out in the interviews and in the documentation review. Thus, while the FV program plays an important role in IC's efforts to meet its commitments to OLMCs,2 this effort is limited by the fact that FV does not address them completely or exclusively.
Because there is a lack of information on the subject, it is not known whether the FV program meets the needs expressed by minority CFAs. Data analysis reveals that the digital divide between Quebec residents and the rest of Canada decreased considerably, owing in part to the FV program. That said, key participants and beneficiaries pointed out other needs that remain, notably ICT training, innovation, and financial and technical support for Web site maintenance.
2 Aside from the Youth Internship Program, FV is the only IC program. There are also other initiatives—regional coordinators (liaison, communications and advising/consulting) as well as distance training and distance education pilot projects.
Audit and Evaluation Branch i Final Evaluation for the Francommunautés Virtuelles Program –Final Report February 2008
The evaluation also shows that the program has evolved in recent years. Increasingly, in order to obtain high scores in the competitive process, applications must be of very high quality. Furthermore, an analysis of the geographic distribution of projects reveals that some regions are under-represented. As a result, inter-provincial equity is no longer assured and the objective of minimizing the digital divide is put on hold in order to pursue more advanced technological goals.
It is difficult to measure the short- and medium-term effect and impact of the projects. Most of the indicators found in final project reports refer to immediate results instead of intermediate and final results. Nonetheless, several results emerge from the lines of inquiry, specifically that the program contributed to the creation of 46 Web sites, which were still active at the time of the evaluation. Funding was also used to create new French-language content in 203 sites. The survey carried out as part of the evaluation shows that more than 3,000 people received ICT training. This suggests that considerable development has taken place in terms of ICT capacity-building in CFAs.
The Francommunautés virtuelles Program has sunset on March 31, 2008. As a result, the evaluation report does not contain any recommendations, and a management action plan from the program was not required. The management response indicated that the program concurs with the findings and conclusions of the evaluation.
Audit and Evaluation Branch ii Final Evaluation for the Francommunautés Virtuelles Program –Final Report February 2008
1.0 INTRODUCTION
This final evaluation was carried out by Groupe-conseil baastel ltée on behalf of the Audit and Evaluation Branch (AEB) of Industry Canada (IC). Its target is the FV program, through which the Government of Canada contributes financially to project delivery in order to meet the specific needs of Francophone communities regarding the use of information and communications technologies (ICTs).
This evaluation focuses on the activities undertaken between 2003 and 2007, but also takes into account all the changes made to the program since its inception in 1998. On the one hand, the evaluation aims to measure the success of the program in terms of whether or not the planned objectives were attained. On the other hand, it provides program managers with the information and objective points of reference necessary to make decisions, while meeting the requirements of the Treasury Board Secretariat (TBS) in terms of performance evaluation. The evaluation also complies with the Evaluation Policy and the Policy on Transfer Payments.
In general, the evaluation should help program managers develop better programs, policies and initiatives from the outset, and periodically provide evaluations of the effectiveness of programs or policies, intentional or unintentional results and other methods that help achieve expected results.
Following the Government committed to the program for a set period of time, the FV program will end on March 31, 2008; therefore, this report is limited to presenting the conclusions drawn from the findings of the evaluation.
Audit and Evaluation Branch 1 Final Evaluation for the Francommunautés Virtuelles Program –Final Report February 2008
2.0 BACKGROUND
2.1 History
The FV program was created in 1998 when the Canadian government introduced a pilot program to provide a more direct response to the ICT needs of Canadian Francophone and Acadian communities (CFAs). At the time, there were significant challenges regarding the use of ICTs by French-speaking people, both within and outside of Quebec. There were gaps in the French-language content on the Internet and CFA use of new ICTs.3
In 2000, the pilot project yielded such positive results in a formative evaluation that the following year, IC and Canadian Heritage (PCH) signed an interdepartmental agreement to follow through with the program as part of the Canadian Culture Online strategy.
In 2002, in compliance with section 41 of the Official Languages Act, the Government of Canada committed to improving the quality of life in Official Language Minority Communities (OLMCs) by promoting their socio-economic and cultural development. The Government also committed to developing an action plan, which was implemented in 2003.
In March 2003, the Government included the FV program in its new Action Plan for Official Languages. In December 2003, a competitive process covering the 2003–2005 fiscal years allowed for the funding of several new projects. Subsequently, the program launched other competitive processes (2005–2007 and 2006–2008), but this time without financial support from PCH.
2.2 Program profile
The FV program is intended to support specific, innovative French-language projects that meet the needs expressed by Canadian CFAs with regard to accessing the information highway in French and acquiring ICT-related skills and knowledge. It is also intended to promote online resource development and networking for remote access and thus contribute to community socio-economic and cultural development with a direct impact on CFAs in Canada.
More specifically, the objectives of the FV program are:
• to encourage the development and use of ICTs in CFAs throughout the country in order to bridge the digital divide in Canada; • to create and promote Web site networks by emphasizing cooperation and partnering within Canadian Francophone and Acadian organizations, aimed at contributing to the socio-economic and cultural development of their communities; and
3 Over the years, several reports have documented the existence of such problems, including the Donald J. Savoie study, Collectivités minoritaires de langues officielles (published in 1998), and the Commission of Official Languages' report, Le français sur Internet : au cœur de l'identité canadienne et de l'économie du savoir (tabled in March 2002). The latter report concluded that there were 16% fewer Francophone Canadians who used the Internet than their Anglophone counterparts and that, according to the trend, this digital divide could widen.
Audit and Evaluation Branch 2 Final Evaluation for the Francommunautés Virtuelles Program –Final Report February 2008
• to develop French-language Web applications, content and services, and make them visible and available through major Canadian portals, including government portals.
The ultimate goals of the program are to promote the participation of Canadian CFAs in the knowledge-based economy; to improve their quality of life and increase their economic growth through learning about and using ICTs; and to strengthen partnering by sharing resources, knowledge, expertise and skills.
In 2005, the National Advisory Board responsible for reviewing funding requests for 2005–2006 decided to take a closer look at the IC mandate to help Canadians become more productive and competitive in the knowledge-based economy, thus improving the standard of living and quality of life in Canada. It therefore asked IC to ensure the program promotes projects that support economic development initiatives.
Since its inception in 1998, the program has supported the delivery of 203 projects, for a total investment of $16,013,987. This report focuses on the last three competitive processes launched by the program, in 2003–2005, 2005–2007 and 2006–2008. For these three competitive processes, the projects submitted were selected by a committee in charge of evaluating them according to a pre-established list of merit criteria (see 2.3.3). In order to be eligible, the applicant had to be a Canadian non-profit organization (NPO), operating for at least one year, with the abilities and skills required to manage and deliver its project. Eligible applicants could receive funding of up to 50% of project costs, up to $75,000 for local and regional projects with a maximum duration of 9 months (component A), or up to $250,000 for national projects with a maximum duration of 14 months (component B). Representing the interests of all French- speaking people, the projects funded had many themes including tourism, trade, economics, education, health, current events and the media. The average amount contributed for the 2003– 2008 period was $101,445, and the total value of funding provided was $9,434,381.
Audit and Evaluation Branch 3 Final Evaluation for the Francommunautés Virtuelles Program –Final Report February 2008
2.3 Program management and governance
The FV program is managed by the Information Highway Applications Branch (IHAB) of IC, which is responsible for its implementation. It is made up of three parts—an Advisory Committee, a program team and a National Selection Committee.
2.3.1 Advisory Committee
An Advisory Committee, which consists of 8 to 12 non-government members, represents all Canadian CFAs so that they can participate fully in community-based economic development (CBED) with respect to ICTs in French.
When required, the Committee provides advice on various aspects of policy development and FV program planning, implementation, evaluation and follow-up. Its members, who are designated by the program, come from all regions of Canada. They ensure that democratic principles are used in decision-making. They are objective, transparent and credible, and they ensure their complete confidentiality in meetings with IC.
The current FV program Advisory Committee is composed of 10 members, who will remain members until the end of the program, scheduled for March 2008.
2.3.2 Program team
IC team members have in-depth knowledge of the FV program and experience in policy development and FV program management. They are involved in the program on a full-time basis, and provide knowledge of the Francophone community environment in Canada. The program team includes seven members —a director, a manager, four project officers4 and a clerk (casual).
IC is required to consider the recommendations made by the Advisory Committee regarding the implementation of the FV Program, and regularly inform the Advisory Committee of its activities. Conversely, the program manager is required to seek advice from the Advisory Committee on various aspects of policy development and program planning, implementation, evaluation and follow-up. Under section 41 of the Official Languages Act, IC must also consult with other departments responsible for official languages in order to promote dialogue among various federal representatives.5
With the help of its Advisory Committee, the program ensures that activities run smoothly and especially that funded projects reflect the needs and realities of communities in terms of developing French-language content, applications and services for the Internet.
4 One development officer is on leave for an indeterminate period and another also acts as team leader. 5 Source: http://francommunautes.ic.gc.ca/comite_2003_mandat_f.asp, consulted on June 28, 2007.
Audit and Evaluation Branch 4 Final Evaluation for the Francommunautés Virtuelles Program –Final Report February 2008
2.3.3 National Selection Committee
The National Selection Committee is responsible for evaluating project proposals according to a pre-established set of merit criteria. It is generally made up of 10 members6 of the Advisory Committee, who accept an offer to join. Furthermore, since the 2003–05 competitive process, the Advisory Committee requires the National Committee include a group of experts whose task is to evaluate the projects. These experts are ICT specialists with excellent knowledge of Canadian CFAs and the realities they face with regard to ICTs. They come from all regions of Canada and are selected for the evaluation period only; they are all non-government members.
2.4 Funding
From 1998–2008, $20 million was invested in CFAs so that they could not only access quality French-language content, but also improve their skills in terms of developing and using advanced technologies. Current program funding will end on March 31, 2008. Information on the $20 million in funding granted since 1998 is provided in Table 1; information on operating and maintenance expenses and salaries is found in Table 2. The number and value of contribution agreements signed following the last three competitive processes appear in Table 3.
Table 1: FV Program Funding from 1998–2008
1998 to 2001 to Period 2003–2004 2004–2005 2005–2006 2006–2007 2007–2008 2000–2001 2002–2003 Funding (millions of 3.0 4.0 1.0 2.0 3.0 3.0 4.0 dollars)
Table 2: Operating and Maintenance Expenses and Salaries
2003– 2004– 2005– 2006– 2007– Period 2004 2005 2006 2007 2008 Operation and maintenance (dollars) 199,667 185,657 197,800 197,000 235,000 Salaries (dollars) 432,193 444,425 472,147 547,000 585,000
6 The number of members depends on needs and the time allocated for evaluating proposals.
Audit and Evaluation Branch 5 Final Evaluation for the Francommunautés Virtuelles Program –Final Report February 2008
Table 3: Number and Value of Contribution Agreements per Competitive Process
Value of Contribution Competitive Proposals Eligible Contribution Agreements Approval Rate Process Received Proposals Agreements Concluded Concluded 2003–2005 154 145 32 22% $2,920,642 2005–2007 71 61 35 57% $3,426,673 2006–2008 101 76 26 34% $3,087,066 Total 326 282 93 33% $9,434,381
2.5 Recent evaluations
Over the past few years, some studies have focussed on the continuous improvement of the FV program. These studies were intended to examine aspects including program rationale, the quality of the design and implementation, progress made, profitability and results. The program was also included in the evaluation of the Canadian Culture Online initiative (led by PCH in 2003), as well as in the evaluation of section 41 of the Official Languages Act and Economic Development of Official Language Minority Communities (EDOLMC) (led by IC in 2005–2006). This is a brief overview of the results obtained in these studies:
• Canadian Culture Online – French-language content on the Internet increased after the program was implemented. However, the production of many reports is a source of dissatisfaction for project beneficiaries.
• Section 41 and EDOLMC – The FV program is generally valued and well-known among OLMC representatives. Two of the positive effects of the program observed were the economic and social leverage it produced and capacity development in OLMC organizations. However, IC managers feel that the relevance of the program has lessened; some question it, while others think it should be continued, provided that it evolves according to the needs of the communities.
Audit and Evaluation Branch 6 Final Evaluation for the Francommunautés Virtuelles Program –Final Report February 2008
3.0 METHODOLOGY AND APPROACH
3.1 Evaluation questions addressed
The evaluation questions addressed in this study can be found in the evaluation matrix in Appendix 1. They concern the relevance, design, delivery, execution, as well as efficiency and cost-effectiveness of the program.
3.2 Evaluation methods
3.2.1 Roles and responsibilities
The mandate was carried out with the participation of the Baastel evaluation team, the evaluation working group and the Evaluation Steering Committee. Appendix 2 lists the members in each group, showing their respective responsibilities in the evaluation process.
3.2.2 Evaluation calendar and strategy
Collection and analysis of the data required for completion of the mandate took place between June and November 2007, in accordance with the final work plan produced by baastel ltée and approved by IC. Whenever possible, the information gathered from different sources was cross-checked in order to base the findings on a solid triangulation of information. For each line of inquiry in the study, the team produced a summary matrix or records covering the evaluation questions in order to compile the results and compare the data from the different sources of information (see case study records in Appendix 3). These elements were used to prepare a list of preliminary findings, and a draft version was sent to the Steering Committee for comments. On the basis of these findings and the comments gathered, the evaluation team produced this report.
3.2.3 Description of the lines of inquiry
Documentation and literature review
For a broader understanding of the program, the evaluation team studied a wide range of documentation, most of which was supplied by IC. This process allowed them to analyse the governance structure and implementation of the FV program, correspondence, analysis and activity reports, the role of different participants, projects funded, anticipated short-, medium- and long-term effects and the results obtained in light of the project reports. In addition, the team conducted an exhaustive literature review to gather a vast amount of information concerning CFAs and their use of ICTs, particularly in light of the reports published during previous, related evaluations. Appendix 4 provides the complete list of documents studied in this phase.
Audit and Evaluation Branch 7 Final Evaluation for the Francommunautés Virtuelles Program –Final Report February 2008
Interviews with key stakeholders
As indicated in Table 4, the team conducted in-person or telephone interviews with a number of key stakeholders. In total, 31 individuals were chosen with the help of IC (see Appendix 5 for the list of participants and Appendix 6 for the two guides used in consultations with them).
Table 4: Number of Individuals Interviewed
Members of the Members of the Other Committee of Advisory Managers Participants Experts Committee 11 10 7 3
Case studies
The evaluation team completed nine in-depth case studies in order to evaluate the benefits of the projects in the community, draw lessons and identify the factors that ensure the viability of projects and the optimization of resources.
Baastel ltée and IC made sure that the sample represented the complete range of projects as much as possible, so that observations drawn from the studies would apply to the entire program. The population of projects to be tested was quite heterogeneous, with 66 projects from two different periods (see Table 5).
Table 5: Characteristics of the Test Population (Case Studies)7
Period 2003– 2005– 2005 2007 Local and regional projects 22 26 National projects 9 9 Total 31 35
The projects differed in scope, province of origin, main sector of activities and level of management and organizational experience of the project promoter. On the basis of all these criteria, baastel ltée and IC chose nine projects for the case studies. A large proportion of these projects (six out of nine) are from 2003–2005; thus, the implementation of the majority of the projects in the cross section is relatively advanced.
7 Projects from the 2006-2008 competition were not included in the test population for the case studies, in order to gain a better perspective of project benefits.
Audit and Evaluation Branch 8 Final Evaluation for the Francommunautés Virtuelles Program –Final Report February 2008
E-surveys
In order to gather the opinion of both FV program beneficiaries and those whose requests were turned down, the evaluation team developed and distributed a survey to 218 organizations that submitted a project in the competitive processes of 2003–2005, 2005–2007 and 2006–2008. After conducting a pre-test with two of the organizations, the team sent a hyperlink by e-mail so that respondents could complete the survey online. The survey was interactive, in that there was a general section as well as visible and invisible sections, depending on the responses of the participants. The survey provided a method for collecting important quantitative data to evaluate certain results achieved by the program and corroborate certain observations from other data collection methods. In total, 92 organizations decided to complete the survey, which corresponds to a 42% response rate.
3.3 Method limitations
The main challenges faced by the evaluation team include:
• Data collection took place over the summer, and so initially there was a very low response rate to the e-mails sent to participants approached for interviews. The team then opted for a more personalized approach: in addition to e-mails, they began making individual phone calls, and this strategy was effective.
• Some members of the Advisory Committee and the committee of experts were unavailable, which prevented some of the planned interviews from taking place. In the end, three interviews were simply cancelled following a consultation with the working group, which decided that the quality and quantity of information already obtained from other interviews was sufficient.
• The lack of documentation on the use of ICTs by CFAs outside Quebec made it impossible to compare the existing situation at the beginning of the program with the current situation. However, various documents will soon be available and may be useful to IC, including the IC socio-economic profiles and the Statistics Canada research report.8
• The survey was launched somewhat later than scheduled because of technological problems and the complexity of the interactive questionnaire. Two reminders were needed, as well as revisions to some e-mail addresses and an extension of the survey period to obtain a satisfactory level of response. Furthermore, the process of data compilation and analysis was arduous because of the length and complexity of the questionnaire. This ultimately weakens the statistical representativity of several survey questions because of the low number of responses. This meant that statistical correlations could not be established for some independent variables (e.g. organization's province of origin), and this affected the entire set of survey questions.
8 Survey on the Vitality of Official-Language Minorities. See:
Audit and Evaluation Branch 9 Final Evaluation for the Francommunautés Virtuelles Program –Final Report February 2008
The strategies used by the evaluation team, including the systematic use of triangulation, enabled them to minimize the effects of these limitations; the team is therefore confident that this report presents valid, reliable information.
Audit and Evaluation Branch 10 Final Evaluation for the Francommunautés Virtuelles Program –Final Report February 2008
4.0 RESULTS
4.1 Relevance
This section primarily covers the program purpose, addressing the following questions:
To what extent is the program compatible with Industry Canada strategic objectives? To what extent does the program play a role in IC efforts to meet its commitments to minority official-language communities under the Official Languages Act? To what extent does the FV meet the expressed needs of Francophone and Acadian communities for ICTs?
4.1.1 Program definition and direction
The program mission and objectives appear to be relevant, however, they are broad and ambitious and do not always point in the same direction. As a result, stakeholdres understand its objectives in a different manner, and this in turn influences their definition of the target population. Indeed, some stakeholders consider it essential that projects funded by the program help their members develop or maintain their competitiveness in a knowledge-based economy, leading to a belief that the FV program must support large, innovative projects managed by ICT specialists. Conversely, other participants stress the importance of bridging the digital divide, i.e. standardizing the use of ICTs throughout the country. In their eyes, the program should support organizations with the least expertise (often small organizations or those in remote or rural areas).9 This dichotomy between innovation and the digital divide causes confusion regarding the target population, which is in turn reflected in differing interpretations of the selection criteria.
The lack of a clear program direction can also be seen in the sphere of operation of the FV program. The selection criteria are not restrictive and can therefore target a heterogeneous population made up of different types of organizations, i.e. community organizations, educational institutions, hospitals, etc. These organizations operate in a wide range of sectors, which are grouped into twelve categories in Figure 1. Given such diversity, not all projects fall within IC's preferred sphere of operation,10 hence the difficulty of IC in justifying the existence of the program. In an attempt to solve this problem, IC insisted in the last two competitive processes that projects respect an economic development criterion, rather than socio-economic, but without clearly defining the desired change. Thus, despite the recommendation of IC to focus on this criterion, most projects did not relate directly to economic development (see Figure 1). Indeed, although it has increased, the proportion of projects classified in the economy sector
9 The IC Web site describes the FV program as follows: “The program is thus able to carry out its mission, which is to position Canadian CFAs to benefit from the digital revolution. The projects funded by the program will contribute to the development of French Canada's communities and economy, and will help French-speaking Canadians to compete in a knowledge-based economy.” The Industry Canada Francommunautés virtuelles program seeks to promote the active participation of Canadian CFAs in the area of developing and using ICTs. The objectives of the Francommunautés virtuelles program are listed in section 2.2. Source:
Audit and Evaluation Branch 11 Final Evaluation for the Francommunautés Virtuelles Program –Final Report February 2008
remains low (5% for 2003–2005, 11% for 2005–2007 and 13% for 2006–2008). This can be explained, first, by the absence of a clear explanation, leaving keyholders to interpret the new directive on their own. Secondly, the project selection method was not changed to give the economic development criterion precedence over the quantitative criteria; the project selection grid assigns it only 30 out of 160 or 180 points (components A and B).11
Figure 1 Distribution of Amounts Awarded, by Sector
Amounts Allocated by Sector (2003–2008)
Tourism
Society
Health
Justice
Informa.and Int. serv.
History
Sector Training
Education Trade and economy
Arts and culture
Agriculture Current events and media
$- $500,000 $1,000,000 $1,500,000 $2,000,000 $2,500,000 $3,000,000 Amount
11 There is also a part reserved for the economic dimension of the project, in the general comments section.
Audit and Evaluation Branch 12 Final Evaluation for the Francommunautés Virtuelles Program –Final Report February 2008
4.1.2 Better identification of winning needs
Owing to a lack of sufficient data on the subject, it is uncertain whether the FV program responds to the needs expressed by CFA minority communities. Therefore, it would be appropriate to review the research and consultations, so as to understand the current ICT situation and discern current needs among the CFA outside Quebec.
The results of some current studies may prove useful, namely those being carried out by the provinces, IC's socio-economic profiles and the Statistics Canada research report on the vitality of official-language minorities. Existing data suggest approaches to the evolution of ICTs in Canada and Quebec.
• According to the IC evaluation of the 2004–2008 Action Plan and the EDOLMC (p. 28), the needs of OLMCs include regional promotion, youth retention and recognition on the part of IC of the economic role of community organizations, as well as easier access for these organizations to IC programs.
• In Quebec, there has been a marked improvement in connectivity and Internet use in the last six years, considerably reducing the digital divide. In fact, the proportion of Quebeckers using the Internet doubled between 2000 and 2006, going from 34% to 68%, essentially attaining the Canadian average.12 According to the Canadian Internet Use Survey (CIUS), 83% of those who looked for information in French said they found it, which shows the accessibility of French content on the Internet.13 According to another study, conducted in November 2006, 71.5% of Quebec adults used the Internet at least once a week, compared to 39.7% in 2000. For the whole year, the proportion reached 65.8 % in Quebec, compared to 67.5% for all of Canada.14
• With respect to the creation of content in French, interviews identified two divergent points of view. Some stakeholders believe that developing content in French has been and will be an ever-present need because French cannot compete with English. Others believe that the digital divide has decreased enough to surpass this objective. The NETendances survey of organizations conducted by the Centre francophone d'informatisation indicates that in 2007, 58% of Quebec Internet users interviewed browsed mainly on French sites, while 17%
12 Moreover, in December 2005, 65% of adults were living in households connected to the Internet (50% high-speed). Quebeckers are using the Internet for new activities, such as employment searches, mailing lists, blogging, online training and telephony. Lastly, 58% navigate mainly in French (as opposed to 48% in 2000). Source: Bisson (2007), pp. 69–72. 13 The proportion of individuals who navigate in French varies according to a number of factors, and not only as a function of the information available in French. For example, it can depend on the user’s preference and habits. 14 However, the study also points out that rate of use varies considerably depending on a number of variables (sex, age, level of education, income, location, first language) and that the rate of use varies considerably from region to region. NETendances annual survey, conducted by the Centre francophone d'informatisation des organisations (CEFRIO), in partnership with Léger Marketing. http://www.directioninformatique.com/di/client/fr/DirectionInformatique/Nouvelles.asp?id=42503, consulted on November 20, 2007.
Audit and Evaluation Branch 13 Final Evaluation for the Francommunautés Virtuelles Program –Final Report February 2008
browsed equally in both French and English. In 2000, fewer than half (47%) habitually used sites in French.15
• Technology has also evolved, becoming more and more accessible. According to a survey of 70 Francophone community organizations,16 the Internet is a tool available to community organizations, because of its reasonable cost and ease of use. The majority of the 70 organizations interviewed invested less than $5,000 for the Web sites. Moreover, this same pan-Canadian study found that, in most cases, creating and updating a Web site is inexpensive and accessible for most organizations. However, some Web sites created at great expense are difficult to maintain today, given the constant evolution of ICTs.17 This need to extend the duration of funding was confirmed by open questions in the survey (desired improvements), case studies and telephone interviews conducted by the evaluation team. Moreover, stakeholders and beneficiaries indicated other needs as well, i.e. training in ICTs, innovation, and financial and technical support for Web site maintenance.
4.1.3 Who does Francommunautés virtuelles target?
In 2003, IC renewed the FV program by integrating it into its Action Plan for Official Languages. Thus, the program not only targets the OLMCs themselves, but also the entire Canadian Francophonie, an inconsistency brought out in interviews and in the documentation review. That said, organizations have different capabilities and needs according to whether they are in a minority or majority situation. While it may be true that the FV program plays an important role in the efforts of IC to meet its commitments to OLMCs,18 these efforts are limited by the fact that FV does not exhaustively or exclusively address them.
Needs are more significant for official-language communities in minority situations than for those in majority situations. In fact, in the interviews, most participants justified the program rationale by listing its impacts on CFAs in minority situations, i.e. ending isolation, promoting the community, networking, being connected with the rest of the country, etc. Given that there is not enough data, it is impossible to draw conclusions about the needs of English-speaking OLMCs or about those more specific to CFAs in minority situations.
15 Annual NETendances survey, conducted by the Centre francophone d'informatisation des organisations (CEFRIO), in partnership with Léger Marketing and Bisson (2007), p. 72. 16 Study conducted by the firm Ronald Bisson and Associates for Francommunautés virtuelles: État des lieux communautaires – Profil organisationnel et financier des sites Internet de la francophonie canadienne à partir d'un échantillon de 70 organismes, Ronald Bisson and Jean Malavoy, February 2006. Source: Bisson (2007), pp. 79–80. 17 Idem. 18 Apart from the Youth Internships program, FV is the only IC program, but other initiatives do exist, i.e. regional coordinators (liaison, communications and advice-counselling) as well as tele-training and tele-learning pilot projects.
Audit and Evaluation Branch 14 Final Evaluation for the Francommunautés Virtuelles Program –Final Report February 2008
4.2 Degree of success in achieving results
This section focuses on questions relating to the benefit and delivery of the FV program, including its accessibility, selection process, governance structure, management of competitive processes and achievement of results. It attempts to answer the following questions:
Has the program contributed to the advantageous positioning of Francophone and Acadian communities in the digital age and in the use of the Internet and ICTs? Have unexpected impacts been observed that are related to the implementation approach? To what extent has the FV program been designed to optimize the objectives and results? To what extent has the FV program succeeded in reaching its audience of Francophone and Acadian communities? To what extent have French-language services offered within the context of projects supported by FV useful to Francophones and Acadians? To what extent has the FV program contributed to the creation, distribution and ongoing use of French-language Web sites and electronic services and applications? To what extent has the program been successful in terms of its leveraging effect on other sources of funding?
4.2.1 Project selection process
A program with vague and unclear contours can still function; however, will achieve better results if some flexibility is built into its implementation. In the case of the FV program, this flexibility has gradually been reduced, weakening the synergy that existed between the participants. This loss of flexibility is explained by a significant change, i.e. up until the 2003–2005 competitive process, the final meeting of the National Selection Committee was chaired by the Advisory Committee representative who first ensured that projects chosen met the criteria and fairness across provinces and territories. Secondly, the representative ensured that applicants had the ability to carry out their projects properly. During this process, committee member exchanges allowed a synergistic dialogue that went beyond the matters assigned, lending a certain leeway to the final choice of projects. This process also provided IC with another lens through which to see the field of ICTs and regional specificities. However, following budget cuts, the process was revised in favour of a simple teleconference, chaired by an external consultant. Since then, discussions have focused exclusively on projects with scores showing a high standard deviation, i.e. projects with quite divergent evaluations. According to countless participants, this new process sidesteps the added value of dialogue, exchange and synergy found in face-to-face meetings. In fact, it increases the burden carried by Selection Committee members, some of whom already find the workload heavy, and all the more so given that they receive little recognition in return.
Selections are made by competent members who represent the different regions and sectors of Canada, and these selections are now almost exclusively based on the project evaluation form that they complete. While some find the form sound, others consider it poorly adapted to the evaluation of project content. The latter report that the consideration given to each of the criteria emphasizes form over substance. Thus, a project can come in with a very low score on the
Audit and Evaluation Branch 15 Final Evaluation for the Francommunautés Virtuelles Program –Final Report February 2008
criterion of meeting the needs of the community, but this can be diluted by the other criteria, allowing the project to ultimately receive a good score that it may not deserve. They add that there are many requirements that are not very useful in terms of project development and implementation, which can limit accessibility to small organizations, for example, because of a single question on the form.
A review of these criteria confirms the risk that a well-founded project presented by a less structured organization may not pass, despite its merits (see Table 6). On this point, interviews, case studies and the survey also indicate a need for assistance in the preparation of applications, by development officers or others.
Table 6: Project Selection Criteria
Specific Quantitative Criteria Points Allocated 1. The manner in which the project will meet community needs. 30 2. The thoroughness of the business plan and the communications plan. 50 3. Planning in terms of human resources and infrastructure. 40 4. The ability to form effective financial partnerships (cash and/or in kind). 30 5. The ability to establish community networks by sharing knowledge and skills Component A:* 20 with the target clientele. Component B:** 40 6. The project benefits on the socio-economic development of the community. 30 Specific Qualitative Criteria Points Allocated 1. The quality (merit) of the project proposal. 20 2. The relevance of letters of support. 30 3. The originality and innovation of the project. 20 4. The feasibility of the project. 20 5. The financial viability. 15 Component A: 305 TOTAL Component B: 325 * Component A: local/regional project, maximum funding of $75,000. ** Component B: national project, maximum funding of $250,000.
The loss of flexibility in the implementation of the program is matched by considerable competition for funding. Together, these phenomena bring about changes in project selection. Now, more than ever before, applications must be flawless to attain a high score. Many small organizations lack the expertise in project submissions that large ones have, which becomes an indirect factor in their exclusion.
4.2.2 Program accessibility
Originally well-designed, the FV program focussed on a rapidly developing sector, ICTs, and responded to the needs of the organizations targeted, i.e. NPOs. That these organizations are often without resources (financial or human) and are trying to survive in an often precarious socio-economic context, contributes to making the design and delivery of convincing projects difficult. As explained in the preceding section, FV is increasingly based on a point system that
Audit and Evaluation Branch 16 Final Evaluation for the Francommunautés Virtuelles Program –Final Report February 2008
isolates more poorly presented projects, creating a disadvantage for organizations with little or no expertise in the preparation and submission of proposals. Moreover, 13 of the 26 non-benficiary organizations (50%) that responded and commented on this subject said that FV did not offer equal opportunity to all types of applicants. These perceptions imply that the constraints facing target organizations can exclude projects that are potentially valid but ineligible owing to applicant deficiencies in substance.
Meanwhile, there are more and more Component B projects, taking up more than 70% of funding distributed in the last competitive process, as shown in Figure 2.
Figure 2 Funding Distribution by Component and Competitive Process
Funding Distribution by Component by Competitive Process
2006–2008 871,011 2,133,048
A 2005–2007 1,623,993 1,844,575 B
2003–2005 1,345,692 1,523,000
0% 20% 40% 60% 80% 100%
It is obvious that only rarely can projects totalling between $150,000 and $500,00019 can be carried out by small organizations, which are often located in rural areas and in minority communities. In fact, out of 37 Component B projects implemented under the program from 2003–2008, only 3 were in rural areas (Paquetville). The major cities where these projects were carried out include Montreal, Ottawa, Quebec City, Halifax and Moncton. The same problem of accessibility for rural organizations is also present for the 56 Component A projects, only 8 of which were in rural areas. Many comments and much data tend to further confirm that there is a sizeable gap in capacities from one organization to another and from one geographic region to another.20 The analysis of project distribution clearly indicates that some regions are under- represented, either because the projects presented are not selected or because no application was
19 FV funds only half of the total cost of projects. 20 What do we wish to evaluate: the “quality” of the applicant or the “quality” of the application?
Audit and Evaluation Branch 17 Final Evaluation for the Francommunautés Virtuelles Program –Final Report February 2008
submitted, which again raises the question of accessibility. Thus from 2003–2008, out of the 93 projects funded, 7 were from New Brunswick, 4 from Manitoba, 3 from British Columbia, 3 from Saskatchewan, 3 from Nova Scotia, 2 from Alberta, 2 from Newfoundland-and-Labrador, 2 from Prince Edward Island, 1 from the Northwest Territories, 1 from Nunavut and 1 from the Yukon, compared to 27 from Ontario and 37 from Quebec (see Table 7). The fact that Quebec receives a large part of project approvals (40%)21 gives rise to a certain discontent on the part of CFAs outside of Quebec, who believe that Quebec organizations have an advantage because they are better structured and equipped to prepare funding applications.
Table 7: Number of Contribution Agreements Granted to Provinces and Territories
Total by Total by Regions 1998–1999 1999–2000 2000–2002 2002–2003 2003–2005 2005–2007 2006–2008 Prov. / Region Terr. BC 1 0 2 1 1 1 1 7 Pacific 10 YT 0 0 1 1 1 0 0 3 MB 1 1 0 2 1 1 2 8 SK 2 1 0 1 3 0 0 7 Prairies and AB 2 1 0 2 1 1 0 7 29 the North NT 2 1 0 0 0 0 1 4 NU 1 1 0 0 0 1 0 3 Ontario ON 14 8 8 8 13 9 5 65 65 Quebec QC 4 4 5 11 8 17 12 61 61 NL 1 0 1 2 0 1 1 6 NS 0 1 1 5 1 1 1 10 Atlantic 38 PE 1 0 1 1 1 1 0 5 NB 3 3 2 2 2 2 3 17 32 21 21 36 32 35 26 203 203 Source: Statistics on the FV program, 1998–2008.
Tighter requirements and accessibility have a greater affect on those provinces and communities with less expertise in drafting project proposals, as well as on certain types of organizations.
Some participants interviewed even call into question the capability of these types of organizations to lead ICT projects on their own. It has been argued that, to improve accessibility by small NPOs, the FV program should unite the public and private sectors more through partnership agreements or even combine knowledge and expertise in project organization teams, i.e. technological partnerships, service partnerships and promotional partnerships.
Furthermore, data collected through all lines of enquiry reveal that application forms are complex and that the heavy burden of reports hinders access and makes implementation more complex. Out of the 47 respondents surveyed who signed contribution agreements with FV and who had an opinion on this subject, almost half (49%) say they had difficulties completing the
21 However, the acceptance rate for Quebec projects is not greater than that of the other provinces.
Audit and Evaluation Branch 18 Final Evaluation for the Francommunautés Virtuelles Program –Final Report February 2008
forms for the application. The proportion is 54% for respondents who did not receive a contribution. Moreover, the requirements are the same for components A and B, while the amounts in question vary considerably.
In addition, several respondents see themselves as “potential instigators” rather than partners, hence the imposition of strict controls, the failure to acknowledge their role as non-governmental organizations and the maintenance of an imbalance between accountability and contribution amounts.
4.2.3 Governance structure
The following figure provides an overview of the governance structure of the program, presented in section 2.3.
Figure 3 FV Governance Structure
Assistant Deputy Minister (Financial Authority) ------Information Highway Applications Branch
Advisory Committee Program National Selection 10 people with Managers and Committee mandates until personnel members members of the March 2008 Advisory Committee & Committee of Experts
The governance structure of the program includes some deficiencies that hinder the effectiveness of its implementation. On the one hand, the mandates of the Advisory Committee (AC) and the committee of experts (CE) are intermixed. Between 1999 and 2002, in addition to being occasionally consulted by telephone, the AC met once or twice a year; its activities then ceased. It lost members, which IC replaced with CE members (who make up the National Selection Committee). Similarly, since the beginning of the program, the AC has been called upon to evaluate projects, adding another role to that of consultation. In short, for a few years, the AC hardly performed its duties any more as an advisory body. It forms part of the National Selection Committee and there is confusion between it, the CE and the selection committee. Yet the duties of the AC are essential and the stakeholders find its members competent and representative of the various regions and sectors of Canada.
Audit and Evaluation Branch 19 Final Evaluation for the Francommunautés Virtuelles Program –Final Report February 2008
Moreover, no regulations specify that a member of the AC or the National Selection Committee cannot submit a project and therefore benefit from the program. This poses no conflict of interest for selection committee members, because they always evaluate proposals from outside of their region. This being said, it is still necessary to ensure that the selection process (requirements, criteria, stages, etc.) be posted clearly enough on the Internet so that committee members have no advantage over the general public. With this in mind, some survey respondents and interviewees recommended that the evaluation grid be made public. However, beneficiaries who developed or thought of developing a project under the program are in a direct conflict of interest if they are members of the Advisory Committee, because they have to give advice on the direction of the program and could confuse their own needs and interests with those of the program.
4.2.4 Competitive process management
Competitive process management is carried out on three levels, i.e. beneficiary organizations, the program and the Department.
External program factors affected FV implementation. First, several stakeholders object to the insufficient time frame between the request for proposals and the deadline for submissions, which they feel limits the creation of true partnerships and the quality of proposals from organizations with no expertise in the matter. Yet, during the last competitive process (2006–2008), there was a time frame of three months, announced one year in advance. While this time frame appears reasonable at first glance, it calls into question the quality of the dissemination of information. Secondly, according to key participants, government funding freezes causes major delays between the announcement of beneficiaries and the awarding of funding, which delays several projects and results in financial losses because funding is not transferable from one fiscal year to another. For example, for the 2005–2007 competitive process, funding was frozen from April to August, obviously holding up the implementation of projects and causing a loss of approximately $1 million. Finally, the high staff turnover within the program may partially explain the program’s lack of precise direction. In fact, since June 2004, the program has had two directors, four managers, four team leaders, three communications officers and seven different project officers.22
With regard to internal management by the program team, opinions varied on many fronts. To begin with, as illustrated in Table 8, respondents to the survey were divided with regard to the clarity of the directives sent, the quality of the follow-up and the support of the program officers. In fact, more than half (56%) of the 39 beneficiaries who commented on the clarity of the directives think that they could be clearer, while 49% of the 43 beneficiaries who commented on the follow-up and support provided by Industry Canada believe these factors should be improved.
The comments made in the survey and the case studies confirm that support and follow-up varied according to the assigned project officer. Moreover, the project officers, having no administrative assistant or secretary, must generally perform administrative tasks themselves— even photocopying. This limits the optimization of resources.
22 Again, the program team includes seven people, i.e. a director, a manager, four project officers and a clerk (casual).
Audit and Evaluation Branch 20 Final Evaluation for the Francommunautés Virtuelles Program –Final Report February 2008
Table 8: Answers from Responding Beneficiaries Who Commented on Directives, Support, and the Attainment of Initial Objectives
Strongly Strongly Question Disagree Agree Total Disagree Agree Did Industry Canada send us clear 10 12 10 7 39 directives? Did Industry Canada provide 13 8 12 10 43 adequate follow-up and support? Did the project meet its initial 18 10 11 8 47 objectives?
The current process includes a budgetary review of projects by officers, who can decrease the allocated amount. According to the survey, 8 of the 39 beneficiaries who answered the question were allocated a budget slightly less than that requested. The organizations that responded thus received an average of $99,046 instead of the average $100,746 requested; therefore, the program officers appeared to make insignificant changes to the proposed budget. Yet, with regard to the management of funds by beneficiary organization, two problems were noted in connection with the amount allocated. To begin with, according to some stakeholders, periods of 9 to 14 months (Components A and B) for the implementation of projects were insufficient, which forced organizations to justify their expenses. Following delays with regard to the initial deadline, the amount is sometimes deemed too large for the time allotted.
Most of the key stakeholders find that funding up to 50% of the project budget involves a reasonable commitment from applicant organizations. However, contributions in kind were questioned, in particular for Component B beneficiaries, whose ability to seek funding is greater than that of more modest organizations. Moreover, to favour organizations with very limited access to other funding sources, some stakeholders think there could be exceptions in the event of extraordinary circumstances when organizations prove that their efforts to obtain funding are in vain and that their project is relevant.
4.2.5 Achieving results
Project performance measurement is directly connected to the logical framework of the program. However, most of the indicators found in final project reports refer more to immediate results than intermediate and final results. This is explained by a total lack of follow-up after the agreement ends. It is therefore difficult, even impossible, to measure the short- and medium-term effects and impacts of the projects. In this regard, the lines of inquiry helped to bring out some important aspects.
Most interview respondents agreed that French-language services offered in projects supported by FV were useful to Francophones and Acadians; the survey confirms this. In fact, out of the 45 respondents whose projects were accepted and who had an opinion, more than half consider that their projects contributed to community economic development and increased the ability of these
Audit and Evaluation Branch 21 Final Evaluation for the Francommunautés Virtuelles Program –Final Report February 2008
communities to participate actively in a knowledge-based economy. The survey provides training data; however, it is difficult to evaluate the number of training sessions given because of the high variability from one respondent to another (the number of training sessions ranged from 1–600). The situation for the number of people trained is similar. Although the gap was quite large from one respondent to another, a total of over 3,000 people received ICT training. This explains the considerable development in terms of ICT capacity building in CFAs.
For their part, the case studies help draw lessons on success factors, difficulties and local benefits (Table 9).
Table 9: Case Studies: Success Factors, Difficulties and Local Repercussions Identified
Success Factors Difficulties Encountered Local Benefits Identified • Establishment of a partnership to • Inability to ensure project • Job creation, especially for the complement the organization's continuity (loss of expertise at duration of the project abilities the end of the project, lack of • Training and capacity building • Prior existence of good resources or expertise for for people involved management abilities updating) • Breaking isolation • Access to experts and good • Need poorly identified • Enrichment for Francophones resources • Turnover among IC officers • Access to products and services • Consistency in relations with the • Heavy workloads for in French (online courses, health FV program beneficiaries, administrative care services, online products, • Good relations with partners, burden tourism, etc.) commitment from players • Respect for deadlines • Creation of networks: highly • Good identification of needs variable among projects • Project self-funding mechanism • With some projects, economic (sale of banners, percentage of growth sales, etc.)
The interviews, documentation review, case studies and survey reveal that from 2003–2008, the FV program has contributed, in some measure, to place CFAs at an advantage in the digital age and in the use of the Internet and ICTs. In fact, the documentation review, case studies and interviews reveal that the program helped increase Internet content in French and improve its quality. FV contributed to the creation and distribution of Web sites, services and electronic applications in French. From the 56 applications accepted, 46 Web sites were created (82%) and were still online at the time of the survey. A program representative confirms that funding was used to create French content for 203 Web sites. The program also contributed to Internet use, albeit indirectly. In fact, the more French services, applications and content there are, the more use there is in French; the more use, the more services are offered. However, the program did not contribute directly to this, because of very poor project promotion outside the network of beneficiaries.
Moreover, the amount of training given and the number of people trained suggests that the program contributed to the development of expertise in ICTs among Canada's Francophone population while partially closing the digital divide between CFAs and Anglophone communities. In fact, according to the Canadian Internet Use Survey (CIUS) that Statistics Canada conducted
Audit and Evaluation Branch 22 Final Evaluation for the Francommunautés Virtuelles Program –Final Report February 2008
in 2005, 83% of Internet users who said that French was their preferred language got the information that they were looking for in the language of their choice. Of those who said that their language of choice was English, 97% said the same.
Several small organizations also made themselves known and increased their visibility. What emerges from the case studies, documentation review and interviews is that the program helped minority French language communities catch up with the pace of modern life, promote themselves, free themselves from isolation and communicate with their community and the general public.
In short, the lines of inquiry helped prove that the program had positive benefits for communities. However, these benefits are not optimal, namely because of some limits with regard to project promotion and continuity.
4.3 Cost-effectiveness
The goal of this final section is to position the program in relation to its major objectives: to develop Internet applications, content and services in French; to encourage the development and use of information and communications technology in Francophone and Acadian communities; and to create networks and partnerships.
This section will answer the following evaluation questions:
• How much did the program contribute to the development and strengthening of partnerships and networks in CFAs? • How much did the program contribute to increasing the ability of the targeted communities to participate actively in a knowledge-based economy? • How efficient and effective was the FV program in its execution?
4.3.1 CFA needs assessment and dissemination of information
To be effective with regard to the objectives and desired results of the program, it is essential for project proponents to know, to the best of their abilities, how to study and assess the needs expressed by the population. The relevance, usefulness and potential use of content, applications and services created directly depend on this.
Information with regard to FV was not disseminated much via existing channels (i.e. the RDÉE, FCFA, CFDC, and regional development agencies) and especially via the Internet, in the east.23 Despite this, most of the people interviewed said that, owing to time and word of mouth, the program is well known in most potential community organizations. However, the interviews and survey indicate a lack of promotion and advertising of the program and developed projects. This may limit knowledge of the program and the use of the content, applications and services created. Of course, unused content has no reason to exist.
23 Some people mentioned finding the program on the Internet by chance, which indicates a lack of dissemination.
Audit and Evaluation Branch 23 Final Evaluation for the Francommunautés Virtuelles Program –Final Report February 2008
4.3.2 Budgetary review
The people interviewed consider Component A relevant and unanimously agree with the $75,000 contribution cap. Some suggest dividing it into two subcomponents, in which one of them would be reserved for small-scale funding ($15,000–20,000) with less onerous administrative requirements in order to ensure accessibility for small organizations. Component B raises serious issues. Opinions differ on the importance stakeholders attach to the various general objectives of the program.24 To those who deem bridging the digital divide a priority, the amount allocated to Component B was excessive, whereas for those who rely on innovation, this component is essential. This difference in opinion aside, many find that the amount allocated is excessive in relation to the products created; others worry that organizations would use it to “survive” and point out that many of the projects carried out were not really “pan-Canadian.”
4.3.3 Creation of networks and partnerships
The program is deemed effective in creating and promoting networks by emphasizing cooperation and partnership within Canadian Francophone and Acadian organizations in order to contribute to the socio-economic and cultural development of their communities. According to the survey, the program seems to generate a number of partnerships, although this varied from one project to another. Table 10 presents the partnerships in the most recent application from organizations that answered the question.
24 For more information, see section 4.1.
Audit and Evaluation Branch 24 Final Evaluation for the Francommunautés Virtuelles Program –Final Report February 2008
Table 10: Number of Partnerships Developed or Strengthened by Category
Partner Categories Respondents Average*** Min. Max. Organizations Private (n*=15) Funding 8 15 1.9 1 4 Collaboration 13 31 2.4 1 10 Dissemination 9 8 0.9 1 2 30** 54 3.6 Public (n=18) Funding 13 15 1.2 1 2 Collaboration 20 65 3.3 1 10 Dissemination 13 76 5.9 1 36 46 156 8.7 Semi-public (n=12) Funding 4 7 1.8 1 3 Collaboration 10 28 2.8 1 10 Dissemination 9 31 3.4 1 10 23 66 5.5 Communities (n=28) Funding 9 35 3.9 1 20 Collaboration 27 477 17.7 1 300 Dissemination 19 321 16.9 1 200 55 833 29.8 Others (n=3) Funding 1 1 1.0 1 1 Collaboration 1 6 6.0 6 6 Dissemination 1 12 12.0 12 12 3 19 6.3 * n refers to the number of sole respondents in one category. ** One organization may have created more than one type of partnership. The same organization may therefore be included more than once in the total. *** The total of the average is calculated from n.
The documentation review confirms this finding: Because of financial partnerships, FV encouraged partnerships and the exchange of resources, knowledge, expertise and skills through the learning and use of ICTs. Another observation includes network strengthening, through the portals and practice communities created.
However, several stakeholders and beneficiaries mention that a significant gap remains between what was planned, the partners' actual commitment, and the depth of these partnerships. It is also noted that IC planned no meeting, forum, symposium, training session or other event to promote networking among the various beneficiaries, despite AC recommendations to this effect at its meeting in Winnipeg in 2005.
Audit and Evaluation Branch 25 Final Evaluation for the Francommunautés Virtuelles Program –Final Report February 2008
5.0 CONCLUSION
The Francommunautés virtuelles program has had positive effects on communities. Nonetheless, there are several grey areas in the definition of FV, and this lack of clarity is reflected in its implementation on various levels.
The program mission and objectives seem relevant, but they do not always point in the same direction. Some participants consider it essential that the projects funded help develop or maintain their competitiveness in a knowledge-based economy, whereas others emphasize the importance of bridging the digital divide. This dichotomy between innovation and the digital divide leads to confusion in the target population.
In 2003, IC renewed the FV program by integrating it into its Action Plan for Official Languages. However, the program targets not only Official Language Minority Communities (OLMCs), but also the entire Francophone population in Canada, an inconsistency pointed out in the interviews and in the documentation review. Thus, while the FV program plays an important role in IC's efforts to meet its commitments to OLMCs,25 this effort is limited by the fact that FV does not address them completely or exclusively.
Because there is a lack of information on the subject, it is not known whether the FV program meets the needs expressed by minority CFAs. Data analysis reveals that the shortcomings of concern to Quebeckers decreased considerably, owing in part to the FV program. That said, key participants and beneficiaries pointed out other needs, notably ICT training, innovation, and financial and technical support for Web site maintenance.
The evaluation also shows that the program has evolved in recent years. Increasingly, in order to obtain high scores, applications must be impeccable. Furthermore, project distribution analysis indicates that some regions are under-represented. As a result, inter-provincial equity is no longer assured and the standardization objective of using ICTs on a Canada-wide scale is on hold in order to pursue more advanced technological goals.
It is difficult to measure the short- and medium-term effect and impact of the projects. Most of the indicators found in final project reports refer to immediate results instead of intermediate and final results. Nonetheless, several results emerge from the lines of inquiry, specifically that the program contributed to the creation of 46 Web sites, which were still active at the time of the evaluation. Funding was also used to add create French-language content to 203 sites. The survey carried out as part of the evaluation shows that more than 3,000 people received ICT training. This suggests that considerable development has taken place in terms of ICT capacity building in CFAs.
The Francommunautés virtuelles Program has sunset on March 31, 2008. As a result, the evaluation report does not contain any recommendations, and a management action plan from the program was not required. The management response indicated that the program concurs with the findings and conclusions of the evaluation.
25 Aside from the Youth Internship Program, FV is the only IC program. There are also other initiatives—regional coordinators (liaison, communications and advising/consulting) as well as distance training and distance education pilot projects.
Audit and Evaluation Branch 26 Final Evaluation for the Francommunautés Virtuelles Program –Final Report February 2008