Evaluation Of The Management Arrangements For The Civil Society Water, Sanitation And Hygiene Fund

Evaluation of the management arrangements for the Civil Society Water, Sanitation and Hygiene Fund 2 Office of Development Effectiveness

October 2016

3 i.Foreword

It is all too easy for evaluations for focus on the negatives: what hasn’t worked well, what needs to be improved, and what should be done away with altogether. We can learn a lot when such issues are highlighted. But learning from the positives – the success stories that provide good practice models – can be even more instructive and useful. It was for this reason that I found the evaluation of the Civil Society Water, Sanitation and Hygiene Fund refreshing and insightful. It highlights the strong performance of the Fund’s management arrangements in areas where many other development programs underperform. The Fund’s approach to knowledge and learning, particularly its face to face learning events, has ensured lessons and knowledge have been identified and shared between civil society organisations. Gender equality and disability inclusiveness have been embedded into the program’s monitoring, evaluation, reporting and learning systems, pushing civil society organisations to do more in these areas. And many aspects of the Fund’s monitoring and evaluation arrangements, such as the use of long-term specialists and the coaching provided to civil society organisations, are highly effective. Such practices provide strong models which other DFAT programs could benefit from. In view of this I think it is important that DFAT managers and staff become fully aware of the positive impacts of the Fund's experience and be prepared to consider employing the Fund approach when and where it is viewed as appropriate.

Jim Adams Chair, Independent Evaluation Committee

Office of Development Effectiveness The Office of Development Effectiveness (ODE) is an independent branch within the Australian Government Department of Foreign Affairs and Trade (DFAT). ODE monitors the Australian aid program’s performance, evaluates its impact, and contributes to international evidence and debate about aid and development effectiveness. ODE’s work is overseen by the Independent Evaluation Committee (IEC), an advisory body that provides independent expert advice on ODE’s evaluation strategy, work plan, analysis and reports. www.ode.dfat.gov.au

4 ii. Contents

Disclaimer: The views contained in this report do not necessarily represent those of the Australian Government.

With the exception of the Commonwealth Coat of Arms and where otherwise noted all material presented in this document is provided under a Creative Commons Attribution 3.0 Australia (http://creativecommons.org/licenses/by/3.0/au/) licence. The details of the relevant licence conditions are available on the Creative Commons website (accessible using the links provided) as is the full legal code for the CC BY 3.0 AU licence (http://creativecommons.org/licenses/by/3.0/au/legalcode). The document must be attributed as Evaluation of the management arrangements for the Civil Society WASH Fund.

5 iii.Executive summary

The Civil Society Water, Sanitation and Hygiene Fund is a six year, $103 million program. It supports 13 civil society organisations (CSOs) to implement 29 water, sanitation and hygiene (WASH) projects in 19 countries across Asia, Africa and the Pacific. This evaluation examines the Fund’s management arrangements. These arrangements are relatively unusual, and relatively resource intensive, compared to many other DFAT-funded programs. They include: Strategic oversight from DFAT’s Water, Sanitation and Hygiene Section (DFAT WSH). This section also holds all major contracts related to the Fund - 17 in total Day to day Fund management by a contracted Fund Management Facility (FMF) A Knowledge and Learning (K&L) component which aims to generate evidence and facilitate learning between Fund CSOs. This component is managed by the Knowledge and Learning Manager (KALM), who is part of the FMF An independent Monitoring, Evaluation and Review Panel (MERP) which supports CSO projects and is responsible for Fund-wide monitoring and evaluation In the countries where Fund projects are implemented DFAT Posts are involved to varying degrees, while CSOs engage with national and sub-national partner governments. The Office of Development Effectiveness (ODE) has conducted the evaluation because of the relative uniqueness of the management arrangements. The evaluation aims to inform the management arrangements for the remainder of the Fund’s life and the design of the Fund’s next phase. The evaluation also considers the competitive selection process through which CSO grants were awarded and suggests improvements for the future. The findings will be of interest and relevance to DFAT WSH, other Fund stakeholders, and other parts of DFAT which manage CSO grant programs. Four areas are outside the evaluation’s scope. It did not examine the effectiveness of CSO projects. Instead, this is being examined by the MERP through their mid-term review documents, which focus on the Fund’s achievements and approaches to date. The evaluation did not compare the Fund’s delivery mechanism (i.e. a centrally managed grants program for CSOs) with other possible WASH delivery mechanisms, as this issue is less relevant for the design of the Fund’s next phase. It also did not consider different options for Fund management, or examine the Fund’s Innovation and Impact Fund. The evaluation used four main methods to examine the management arrangements and CSO selection process. First, interviews with key stakeholders were conducted, including DFAT WSH, FMF staff, MERP members, and CSO representatives. Second, an online survey of in-country CSO teams was conducted. This survey was also provided to CSOs who applied for the Fund but were unsuccessful. Third, a review of key Fund documents was conducted. Finally, the Fund was compared to two similar DFAT-funded CSO programs: the Australian NGO Cooperation Program (ANCP) and the Australia-Africa Community Engagement Scheme (AACES). This comparison drew heavily on recent evaluations of these programs. The evaluation’s recommendations were developed through a collaborative process involving the evaluation team, DFAT WSH, the FMF and the MERP.

Evaluation findings: Fund management arrangements The evaluation finds that the Fund’s management arrangements are largely appropriate. The management arrangements are consistent with the broad consensus in DFAT on the elements needed to manage a CSO grants program. They address needs that were identified in reviews of the Fund’s predecessor program. The evaluation also finds that the Fund’s management arrangements are functioning well and support CSOs to implement high quality WASH projects. The different groups in the management team are largely perceived to work in a collaborative, integrated manner which is helpful for CSOs. The evaluation found the management arrangements have a number of strengths. These provide a strong model which other DFAT programs could learn from and should consider adopting and adapting to suit their needs. The strengths include: The Fund's focus on knowledge and learning. CSOs commented that, unlike some other donors, DFAT has supported its commitment to knowledge and learning with appropriate resources. The Knowledge and Learning component, and particularly its face to face learning events, has supported CSOs to improve their projects. 15 of 16 survey respondents stated that learning events had improved their project team’s WASH knowledge and practices either ‘significantly’ or ‘extremely’ Elements of the Fund’s performance assessment arrangements. This particularly includes the MERP model of having monitoring and evaluation (M&E) experts engaged for the life of the Fund, the MERP support provided to CSOs through monitoring visits and the flexibility given to CSOs to adapt and change their projects as they learn during implementation The Fund’s approach to gender equality and disability inclusiveness. The Fund’s M&E arrangements have ensured CSOs design, implement and report on gender equality and disability inclusiveness. CSOs have been supported to improve their approaches to these issues through advice from the MERP and through knowledge and learning events. As a result, a number of CSOs reflected that they have been pushed to go further in these areas than they might otherwise have done. The engagement between CSOs and partner governments in countries where projects are implemented is also a strength of the Fund’s management arrangements. Although the Fund's management arrangements are, overall, functioning well, there is room for improvement in four areas. First, the management of CSO grant agreements could be more efficient. DFAT WSH holds the grant agreements with CSOs, yet the FMF is tasked with the day to day management of the grant agreements. This can create double handling and additional work for both DFAT WSH and the FMF. An alternative arrangement where the FMF holds the grant agreements with CSOs could be more efficient. Second, the Fund’s reporting arrangements could be improved. In relation to project-specific reporting, CSO views are mixed. Some CSOs are positive about the reporting requirements, while others feel reporting is too compliance focussed and does not provide space for high level learning and reflection. The evaluation finds that the intended purpose of project reporting, being accountability to DFAT, is appropriate and necessary to ensure Fund-wide results can be reported. The evaluation also finds that there is a mismatch between this intended purpose and some CSOs’ preferences for reporting focussed on learning. The evaluation strongly supports the MERP’s planned review of the CSO project reporting system and suggests it should actively consider whether the system could be amended to cover both accountability to DFAT and facilitation of CSO learning and reflection. In relation to Fund-wide aggregate reporting, CSOs and DFAT WSH expressed concern that such reporting may not meet their needs. CSOs feel that evidence-based lessons from the Fund have not been available to them and that effective ‘feedback’ loops to help them understand evidence emerging from the Fund do not exist. DFAT WSH similarly has concerns that Fund-wide aggregate reporting is resource intensive without necessarily meeting their needs. The evaluation finds that DFAT WSH has found it challenging to define what they require from the Fund’s performance system and should devote more resources to engaging with strategic level M&E issues. The third area where there is room for improvement is the independence of the MERP. The MERP is contracted directly to DFAT WSH and so is considered independent from the FMF and CSOs. This has benefits for DFAT WSH, who can be assured that Fund data provided by the MERP is credible. However, the majority of CSOs perceive the MERP’s independence has few benefits for them. The MERP’s independence also has disadvantages: it creates additional administrative work for DFAT WSH, it creates risks for coherent management of CSO projects, and it does not manage the potential risk of dysfunction within the Fund management team. The evaluation finds that, for the Fund’s next phase, the primary factor in determining the MERP’s contracting arrangements should be ensuring the best possible team is put in place, rather than questions of independence. Fourth, the evaluation finds that engagement between the Fund and DFAT country programs (particularly in-country Posts) is variable and, in some cases, less than ideal. This is a common issue for centrally-managed DFAT programs. All parties (DFAT WSH, Posts and CSOs) agree that greater engagement would be beneficial. Broad DFAT experience demonstrates that staff at Post have competing priorities to manage, making it incumbent on DFAT WSH to place higher priority on managing relationships with country programs. The Fund’s management arrangements are working well, but they are relatively high cost compared to two other DFAT funded CSO programs, ANCP and AACES. The evaluation found that: The Fund has high management costs when compared to ANCP, but the Fund’s management arrangements are also more effective than ANCP’s at the time it was evaluated AACES focussed on equal partnership between CSOs and DFAT. This resulted in management arrangements that had a similar level of effectiveness to the Fund’s management arrangements. However, AACES’ partnership approach had lower contract costs – noting that much of the monitoring and evaluation support and knowledge and learning elements in AACES were undertaken by CSOs, meaning the costs of such functions are hidden when examining contract costs. Although the Fund’s costs are higher than other programs, they are justified because of the effectiveness of the Fund’s management arrangements and the value placed by CSOs on elements such as the knowledge and learning component. For future Fund phases, the Fund would benefit from systems which can be used to justify its relatively high management costs.

Recommendation 1

For the Fund’s next phase, the Fund’s monitoring and evaluation system includes arrangements which ensure the Fund’s management costs can be justified. To help inform the design of the Fund’s next phase, the evaluation team considered whether the Fund’s management arrangements are relevant to the changing context of the Australian aid program. This includes constrained resources (particularly staff resources), reduced aid management expertise, a greater focus on public diplomacy, and the fact that the Fund represents a greater proportion of Australia’s WASH funding now when compared to the Fund’s commencement. Taking this context into account, it is the evaluation team’s judgement that the Fund’s approach to management will remain relevant for the Fund’s next phase and that all key elements should be retained. Further, DFAT WSH should focus its scarce resources on areas where it can add the greatest value: strategic oversight (including for monitoring and evaluation), relationships with country programs and public diplomacy. In light of this, the evaluation team finds it is appropriate for CSO funding agreements to come under a future FMF. It is also appropriate for DFAT WSH to contract expertise that is scarce in the department, in areas such as monitoring and evaluation and knowledge and learning.

Recommendation 2

That for the Fund’s next phase, DFAT WSH commits its resources to areas where it has the greatest value add, particularly strategic oversight (including for M&E and reporting), relationships with country programs and public diplomacy. At the same time, DFAT needs to ensure it maintains strong relationships with Fund CSOs, that all parties to the Fund have a forum where management issues (such as Fund reporting) can be discussed, and that ‘feedback’ loops are created so that CSOs are aware of Fund-wide reporting and evidence. For the Fund’s next phase, this could be achieved through a Fund-specific advisory or consultative group that includes DFAT WSH, the FMF, the MERP and CSOs. The exact remit of the group, as well as its intersection with other groups such as the WASH Reference Group, should be further explored during the design of the Fund’s next phase.

Recommendation 3

When designing the Fund’s next phase, the design team considers a Fund-specific advisory or consultative group which includes all members of the Fund management team and Fund CSOs.

Evaluation findings: CSO selection The Fund used a competitive grants process to determine which CSOs and projects to support. Selection was in two phases: A selection panel made up of staff from DFAT WSH, the DFAT NGO section, and an external WASH consultant assessed applications submitted by CSOs 13 CSOs were chosen by the selection panel to proceed to a three month ‘inception phase’. In this phase CSOs were provided funding and MERP coaching to develop project designs, known as Operational Plans. Once Operational Plans were approved by DFAT WSH, CSOs were invited to enter into a full funding agreement covering the life of their project(s). The selection process had many strong elements, such as its staged approach, the diversity of the selection panel, and the weight given to gender equality and social inclusion. It resulted in the selection of a number of high quality CSO projects. However, stakeholders perceive that approximately 20 per cent of the projects selected do not align with the Fund’s design intent or theory of change. Such projects are seen to focus on the direct delivery of WASH services, rather than focussing on achieving sustainable change by working on the WASH enabling environment. The evaluation finds several factors may have contributed to the selection of such projects: The Fund’s theory of change was not strongly articulated until after CSOs applied for the Fund, meaning CSOs did not have had a complete picture of the Fund’s intent Important selection criteria, such as those related to a CSO’s understanding of sustainable WASH approaches, were not given sufficient weight in the selection process CSO past experience within the WASH sector and within the previous Fund (where applicable) was not an intrinsic part of assessing that CSO’s claims The diversity of the selection panel was not used to its full potential, with project proposals examined by either WASH experts or NGO experts, and The expertise of the external WASH consultant may not have been appropriate for the types of projects the Fund was aiming to support.

Recommendation 4

That for future CSO selection, consistency with the Fund’s intent and theory of change is a heavily weighted criteria. Recommendation 5

That for future CSO selection, CSO performance in the WASH sector and the current Fund is explicitly taken into account.

Recommendation 6

That for future CSO selection, panel members have appropriate expertise (in WASH, gender equality and disability inclusiveness, and with CSOs). A number of other areas also warrant further consideration in the Fund’s next phase. DFAT WSH did not have sufficient human resources to manage the selection process, suggesting greater resources, as well as use of an online grants management system, should be considered. Given the relatively small number of CSO applicants for the Pacific for the current Fund, DFAT WSH may also want to consider the available pool of applicants for the next Fund and how this might impact on CSO selection. Finally, the size of the grants that were awarded were smaller than anticipated in the Fund’s design and grant size could be reconsidered in consultations with CSOs for the Fund’s next phase. The evaluation finds the Fund’s inception phase was beneficial and should be retained. The inception phase was universally praised by Fund stakeholders. CSOs felt it allowed them to improve the quality of their Operational Plans and it also gave DFAT WSH confidence that CSOs’ Operational Plans covered all relevant issues. However, an appropriate length of time for the inception phase should be considered further by DFAT WSH. A longer inception phase for the Fund’s next phase may be warranted, particularly if the next phase takes a particularly innovative direction. The evaluation also finds it was not useful for the inception phase to be a ‘stop-go’ point for CSO projects and this feature should not be retained in the future. iv. Management response

The ODE Evaluation has provided valuable insights into the management arrangements of the Civil Society WASH Fund (CS WASH Fund) including the selection process for civil society organisations at the commencement of the Fund. The recommendations reflect extensive discussions with stakeholders and it is clear their input has been carefully considered. The recommendations are helpful in being forward looking and feed very well into the design process for the new initiative, Water for Women, which will follow-on from the CS WASH Fund. In addition, while the recommendations point to improvements for the new initiative, they also lead the DFAT Water, Sanitation and Hygiene Section (WSH) to some adjustments in the current CS WASH Fund. One of the immediate implications of the ODE Evaluation is in the development of the Terms of Reference for the design of Water for Women. In total, the recommendations in this evaluation will result in the Terms of Reference giving particular focus to value for money, the strategic role of DFAT WSH, the governance arrangements and the civil society organisation (CSO) selection process. The Water for Women Investment Concept Note, recently prepared, has also been guided by the ODE Evaluation findings as will the ensuing discussions at the Aid Investment Committee. All recommendations have been agreed and actions identified with timelines where possible. Individual Management response to the recommendations

Recommendation Response Recommendation 1 Agree For the Fund’s next phase, the Fund’s monitoring and evaluation system includes arrangements which ensure the Fund’s management costs can be justified.

Recommendation 2 Agree That for the Fund’s next phase, DFAT WSH commits its resources to areas where it has the greatest value add, particularly strategic oversight (including for M&E and reporting), relationships with country programs and public diplomacy. Recommendation 3 Agree When designing the Fund’s next phase, the design team considers a Fund-specific advisory or consultative group which includes all members of the Fund management team and Fund CSOs. Recommendation 4 Agree That for future CSO selection, consistency with the Fund’s intent and theory of change is a heavily weighted criteria.

Recommendation 5 Agree That for future CSO selection, CSO performance in the WASH sector and the current Fund is explicitly taken into account.

Recommendation 6 Agree That for future CSO selection, panel members have appropriate expertise (in WASH, gender equality and disability inclusiveness, and with CSOs).

1 Introduction: Civil Society WASH Fund

1.1 Background to the Fund The Civil Society Water, Sanitation and Hygiene Fund (the Fund) is a $103 million program that runs over 2012-2018. Its objective is to enhance the health and quality of life of the poor and vulnerable by improving sustainable access to safe water, sanitation and hygiene. The design of the Fund drew heavily on its predecessor program, also known as the Civil Society WASH Fund (2010-2012), and a mid-term review of that program. The Fund is a competitive grants program for civil society organisations. It supports 13 civil society organisations (CSOs) to deliver 29 WASH projects in 19 countries (see Annex A for a summary of CSOs and project locations). Nine projects operate in East Asia, eight projects are in South Asia, and there six projects each for Africa and the Pacific. The Fund is expected to directly benefit up to 3.5 million people and indirectly benefit 10 million people. The Fund is strongly focused on sustainability and the enabling environment. To this end, the Fund’s theory of change outlines that CSOs should influence WASH sector ‘change agents’ (typically government or community actors) who will then deliver or support improved WASH services. The Fund also has a strong emphasis on a number of cross cutting development issues, such as gender equality and disability inclusion. The Fund has six outcomes: Outcome 1: Improved performance of actors in the WASH enabling environment Outcome 2: Improved gender equality Outcome 3: Improved WASH evidence and knowledge base Outcome 4: Improved hygiene behaviour Outcome 5: Increased use of equitable sanitation services Outcome 6: Increased use of improved and equitable water supply services.

1.2 Fund management arrangements The Fund’s management arrangements aim to support CSOs to implement high quality WASH projects. Figure 1 outlines the main stakeholders involved in Fund management. In summary: The DFAT Water, Sanitation and Hygiene Section (DFAT WSH) provides strategic oversight of the Fund. It also holds and manages contracts with all the major stakeholders in the Fund – the Fund Management Facility, the Monitoring, Evaluation and Review Panel (MERP) and the CSOs The Fund Management Facility (FMF), through the Fund Manager, is responsible for day to day Fund management and ensuring smooth communication flows between all parties. The FMF is also responsible for the Fund’s Knowledge and Learning component (described below) The Monitoring, Evaluation and Review Panel (the MERP) is a three member team responsible for the Fund’s monitoring and evaluation arrangements (described below) DFAT Posts in the countries where the Fund is implemented are engaged to varying degrees. Similar to other centrally-managed DFAT programs, Posts are not obliged to be involved in Fund management and can engage to the degree that corresponds with their level of interest Within CSOs who are funded, two levels of staff are involved in fund management. Each of the 29 projects has an in-country project manager who oversees project implementation. CSOs also have a program manager, generally based in CSO headquarters, who oversees that organisation’s portfolio of projects. Note that throughout this report, the main stakeholders involved in Fund management (DFAT WSH, FMF and the MERP) are collectively referred to as the Fund management team. Figure 1: Stakeholders involved in Fund management 1.3 Fund monitoring and evaluation, and learning The Fund has a strong focus on generating evidence-based knowledge, learning and innovation. This is achieved in three ways: CSOs are expected to innovate and share evidence-based knowledge through their projects The MERP supports ongoing learning and application of evidence-based knowledge across the Fund. Further information on the MERP’s role is below A specific Knowledge and Learning component, managed by the Knowledge and Learning Manager from the Fund Management Facility, supports strategic research, innovation and evaluation and the sharing of knowledge and learning across Fund CSO projects and with the wider WASH sector. Further information on this is below.

1.3.1 Monitoring, Evaluation and Review Panel (MERP) The Fund has a three member Monitoring, Evaluation and Review Panel. In the previous Fund a similar panel was viewed by DFAT and CSOs as highly successful, and so the model was retained for the current Fund. The MERP has three main functions. The first is to work in partnership with CSOs to improve CSO projects by working as M&E ‘coaches’. This is achieved by conducting monitoring visits to project sites to support CSOs to identify project improvements1 and by assisting CSOs with project M&E and reporting. Second, the MERP plays a role to support learning across the Fund. It does this by, for example, having a Fund-wide perspective and facilitating the transfer of ideas and innovations across the Fund, particularly through monitoring visits. The MERP also supports the Knowledge and Learning component. Finally, the MERP is responsible for overall Fund monitoring, evaluation and reporting. The MERP designed and oversees the Fund’s performance assessment arrangements, assists CSOs to report against these arrangements, and completes Fund-wide aggregate reporting. As part of this, the MERP is conducting a mid-term review of the Fund’s achievements and effectiveness to date. Nine separate mid-term review ‘notes’ have been produced, covering topics such as an analysis of progress, a discussion of expected changes and emerging impacts, and an outline of approaches to gender equality and social inclusion.

1.3.2 Knowledge and Learning Component One of the recommendations from the completion review of the Fund’s predecessor program was that an enhanced Knowledge and Learning (K&L) component should be included in the Fund’s design. This was adopted by the Fund’s designers and represents a key part of the Fund’s approach to knowledge and learning. As noted above, the K&L component is managed by the Knowledge and Learning Manager (KALM). There are two main areas of work under this component. The first is generating evidence-based knowledge in the WASH sector. To achieve this, the KALM works with CSOs to monitor and share the evidence base developed through Fund projects. This includes management of the Innovation and Impact Fund (I&I Fund) which encourages CSOs to trial

1 As of September 2016 the MERP had conducted monitoring visits to 27 of the Fund’s 29 projects. Two projects in Bangladesh had not been visited for security reasons. new models and approaches to influence the WASH sector. DFAT has also sponsored WASH research projects through the Australian Research Development Awards (ADRAs). One ad-hoc piece of research on how CSOs learn has also been commissioned. Second, the Knowledge and Learning component aims to facilitate the sharing of knowledge and learning between Fund CSOs and with the wider WASH sector. This is achieved in a number of ways including: An interactive website that supports online discussion and webinars (see http://www.cswashfund.org/) Face to face WASH regional and global learning events which bring together CSOs supported by the Fund. Four such events (one global event at inception and three regional events) have already been held and a further two are planned Ensuring evidence generated through the Fund is available to CSOs and the broader WASH sector, for example through the website and participation in global WASH forums, and Facilitating collaboration between CSOs. Note that the MERP and the KALM work closely together on Fund M&E and K&L. For example, the KALM assists the MERP’s work with CSOs by reviewing the K&L sections of CSO project reports and participating in selected monitoring visits. The MERP supports the KALM by, for example, identifying topics which could be covered in learning events and actively participating in these events.

1.4 Fund budget The total budget for the Fund is approximately $103 million. The breakdown of funding is shown in Table 1. The total management costs for the Fund are approximately $8 million, or 7.9 per cent of the Fund’s total. This includes the cost of the Fund Management Facility and the MERP, as well as some small costs incurred by DFAT WSH to set up the Fund. Note that two sets of management costs are not specified in Table 1: The 1.5 full-time equivalent staff which DFAT WSH commits to Fund management CSO-specific management costs associated with each project. The budget for program management is slightly less than the $8.6 million anticipated in the Fund’s design document. Table 1: Budget breakdown across Fund activities2 Total funding % of total CSO grants $94,013,763 91.3 Grants: 92.1% of Fund total CSO grants for Innovation and Impact $793,951 0.77 Fund Fund management administered by $5,263,449 5.11 Management costs: Palladium3 7.9% of Fund total Fund management administered by $135,701 0.13 DFAT4 MERP $2,769,626 2.69 TOTAL $102,976,490 100

1.5 CSO selection and contracting The Fund used a competitive grants process to determine which CSOs and projects to support. Selection was a two-part process. First, Australian and non-Australian CSOs working in WASH in the Fund’s target regions were invited to submit applications. A selection panel made up of staff from DFAT WSH, the DFAT NGO section, and an external WASH consultant assessed the applications. This initial assessment had a number of steps. For example, each CSO had to submit an organisational capacity statement. If this statement met a certain minimum standard the project-specific concepts submitted by the CSOs were then assessed. The 13 successful applicants from the first stage then entered the second stage, known as the ‘inception phase’. The purpose of this three month phase was for CSOs to develop detailed project designs, known as Operational Plans. CSOs and DFAT WSH entered into short term funding agreements, ensuring CSOs were provided with budget for their design work. An inception phase workshop was held in Canberra, bringing together CSOs, the FMF, the MERP and relevant parts of DFAT to support CSO design work. Once Operational Plans were approved by DFAT WSH, CSOs were invited to enter into a full funding agreement covering the life of their project(s).

2 Note MERP contracts were amended in September 2016. As a result this table has been updated when compared to earlier evaluation products. 3 This includes all costs associated with the Knowledge and Learning Component, such as the costs of learning events and website set up and maintenance. 4 This includes work such as the Fund design, selection and procurement processes and the inception phase workshop. 2 Introduction: The evaluation

1. Evaluation purpose, audience and scope The Fund’s management arrangements are relatively unusual, and relatively resource intensive, when compared to other DFAT funded programs. An independent MERP with broad and significant responsibilities such as designing the Fund’s performance and reporting system, providing support to CSOs, and assessing overall Fund performance is not a common feature. A specific Knowledge and Learning component is also unusual, as is a contracting arrangement where DFAT holds the Fund’s contracts but an FMF does much of the day to day work. The main purpose of this evaluation is to examine whether the Fund management arrangements are supporting the Fund as intended, and whether their costs are appropriate. A secondary focus is an analysis of the Fund’s selection of CSO projects and implementation of the Fund’s inception phase. The evaluation’s findings will inform management of the current Fund and the design of the Fund’s next phase, taking into account the changing context of DFAT and the Australian aid program. DFAT WSH section is the main audience for the evaluation. As the overall manager of the Fund, DFAT WSH will make decisions about the current Fund’s management arrangements, a future Fund’s design and CSO selection under that Fund. The FMF and the MERP are important audiences to action recommendations regarding the Fund’s management arrangements. CSOs and Posts currently participating in the Fund should find this evaluation useful. The evaluation also aims to identify broad lessons on the management arrangements which are useful for a wider DFAT audience. Note that this evaluation focussed on a single DFAT investment and so is considered an operational evaluation. It was conducted in a short timeframe by a two member evaluation team. As the evaluation had limited resources it focussed on the Fund’s existing management arrangement and how these could be improved. The following four areas were outside the evaluation’s scope: The evaluation did not examine the effectiveness of CSO projects. This is being examined by the MERP through their mid-term review documents which focus on the Fund’s achievements and approaches to date (see section 1.3.1) The evaluation did not consider alternative delivery mechanisms that could be used by DFAT WSH to deliver WASH funding. It was assumed that the Fund’s current delivery mechanism (a centrally managed grants program for CSOs) would continue for future phases, making an examination of current management arrangements most relevant for DFAT WSH’s planning The evaluation did not consider other significantly different options for the Fund management, or compare the advantages and disadvantages of the current management arrangements with other management options The evaluation did not examine the Innovation and Impact Fund (see section 1.3.2), as the Fund’s broader management arrangements and CSO selection were higher priority focus areas.

2. Evaluation questions The priority evaluation questions are at Table 2.

Table 2: Evaluation Questions 1. To what extent are the Fund’s management arrangements appropriate, effective and efficient?

(i) To what extent are DFAT staff in Canberra and at posts, Fund Management Facility staff, MERP members, and CSO staff working effectively together to achieve Fund objectives? (ii) To what extent are contracting arrangements facilitating the effective and efficient attainment of Fund objectives? (iii) To what extent have the M&E and learning approaches inherent in the Fund’s design and implementation assisted to develop and achieve better outcomes? (iv) To what extent is Fund management cost-effective? (v) To what extent do the Fund’s management arrangements give due regard to advancing equity and inclusiveness, particularly in relation to gender and disability? 2. To what extent did the two-part CSO selection process enable selection of the most effective CSOs to deliver WASH projects? (i) To what extent was CSO selection (including the inception phase) consistent with the design’s guidance and intent? (ii) To what extent did CSO selection give due regard to advancing equity and inclusiveness, particularly in relation to gender and disability?5

3. Summary of main methods The main approach taken in the evaluation was to define the behaviours and communication that a reasonable observer would expect to see if the Fund’s management arrangements support CSOs to deliver high quality projects. The expected behaviours and communication flows are shown in Annex B, which the evaluation team developed in consultation with DFAT WSH, the FMF and the MERP. Data was then gathered to assess the extent to which these behaviours and communication flows are being exhibited. The evaluation team collected data using four main methods: document review, interviews, an online survey, and comparison of the Fund with two other DFAT-funded CSO grant programs. Table 3 summarises the evaluation team’s main data collection sources. In all, the evaluation team reviewed 30 documents and interviewed 41 people through 36 interviews. A planned visit to observe a Fund regional learning event did not occur due to problems obtaining a visa. Annex C provides a fuller explanation of the evaluation’s methods. All evidence used to formulate evaluation findings was triangulated where feasible to ensure a degree of rigour. The main evaluation findings drew on multiple data sources, where possible, as well as being tested through interviews with Fund stakeholders. The evaluation team circulated draft evaluation products to the Fund management team to ensure evaluation findings would be useful. ODE’s three member Independent Evaluation Committee (IEC) also provided comments on the team’s draft products. The evaluation tested main findings with the Fund management team at a recommendations workshop. The workshop allowed the evaluation team and the Fund management team to develop recommendations that flow from the evaluation’s findings and to ensure these recommendations are practical, feasible, and supported by the Fund management team.

Table 3: Data Collection Summary Data Collection Method Stakeholders/ Data

5 Note the evaluation focussed on gender equality and disability inclusiveness in the Fund as these are high priority issues for both DFAT and the Office of Development Effectiveness. Resource constraints meant other cross issues considered in the Fund, such as environment, climate change and disaster risk reduction, were not examined by the evaluation team. Interviews Fund management team DFAT WSH (current and former staff). FMF staff (current and former FM, KALM, program coordinator). MERP members. CSOs 13 CSO program managers, two former CSO program managers, and two CSO staff in supervisory roles. DFAT Posts 5 posts out of 17 participating in the Fund were chosen for interview. One post was selected from each geographic region (Africa, South Asia, South East Asia and the Pacific). The sample included posts with large and with small bilateral WASH programs, and those that had been more actively and less actively engaged with the Fund. Online Survey Distributed to 29 CSO in-country project managers and 159 CSOs that were unsuccessful in obtaining Fund funding. 18 responses received from CSO locally-based project managers (62 per cent response rate).

11 responses received from unsuccessful CSOs (7 per cent response rate). Note that it was not compulsory to answer all survey questions, meaning some questions had fewer respondents than the total number of survey respondents. Documents Documents included: DFAT Aid Quality Checks, contracts, DFAT minutes regarding CSO selection and selection panel documents, Fund reporting by the FMF, mid-term review notes by the MERP, examination of the Fund’s website and online database, design document for current Fund, mid-term and independent completion reports of previous Fund, ODE’s ANCP evaluation (2015) and final evaluation of AACES program (2016).

2.3.1 Comparison with other DFAT-funded CSO projects As noted above, the methods used in this evaluation included a comparison of the Fund with two other DFAT-funded CSO projects: the Australia Africa Community Engagement Scheme (AACES) and the Australian NGO Cooperation Program (ANCP). Both AACES and ANCP support CSOs to implement development activities. Under AACES, DFAT partnered with ten Australian NGOs to deliver food security, maternal and child health, and WASH activities in 11 African countries. AACES received $83m funding over five years (2011-2016). The ANCP is an annual grants program that provides funding to accredited Australian NGOs to support work in developing countries. It is DFAT’s largest program for NGOs, expending approximately $130m per year. It supports 53 Australian NGOs to implement 575 projects in 57 countries.6 ANCP projects focus on a broad range of sectors including health, WASH, governance, education and agriculture. Both AACES and ANCP have recently been independently evaluated. The ANCP was evaluated by ODE in 2014-15, while an independent end-of-program review for AACES was completed in 2016. This evaluation draws heavily on the evaluations of AACES and ANCP to: Identify common management features in DFAT-funded CSO programs, and Benchmark the Fund’s management costs, as well as the effectiveness of its management arrangements, against ANCP and AACES to determine whether the Fund is cost- effective.

2.4 Limitations While the evaluation team believes that the methodology and process are appropriate to inform future Fund management and the design of a new Fund, there are a number of limitations. These include:

6 See http://dfat.gov.au/about-us/publications/aid/Pages/the-australian-ngo-cooperation-program-fact-sheet.aspx Focus on Fund processes and not on Fund outcomes: As the evaluation did not assess the effectiveness of CSO projects, or progress towards the achievement of the Fund’s six outcomes, it is unable to independently draw links between the Fund’s management arrangements, the quality of CSO projects and the achievement of Fund outcomes. To better understand the Fund’s progress against intended outcomes, the evaluation team was informed by the MERP’s nine mid-term review notes finalised in September 2016. However, the evaluation team was not able to independently verify the MERP’s work through this evaluation. In relation to CSO selection, the evaluation team makes no judgement on the track record of CSOs selected. The team suggests process-level improvements for CSO selection, rather than judgments on the merits of CSOs’ WASH expertise or the effectiveness of their projects. Sampling of data: The evaluation was conducted in a relatively short timeframe to ensure findings would be available to inform the design of the Fund’s next phase. As a result, interviewees were heavily prioritised and some stakeholders were not interviewed. These included external stakeholders who may have views on the Fund’s engagement and influence in the broader WASH sector and local ‘change agents’ whom CSOs work with. The evaluation team also did not interview the external consultant on the CSO selection panel, instead relying on a document review to analyse differences in panel members’ ratings of CSO applications. Low response rate to survey: The CSO survey received a low response rate from CSOs unsuccessful in obtaining Fund support. It is possible there is a large non-response bias (that is, those who responded to the survey may differ in significant ways from those who did not respond). This constrained analysis of non-successful CSO views on CSO selection. Comparison of the Fund with AACES and ANCP: The comparison relied heavily on the existing evaluations of these other programs as the evaluation team was not able to collect primary data on them. Although the evaluation team compared the programs’ different management arrangements, it was not possible to conduct a detailed analysis of the advantages and disadvantages of the different management approaches taken by the different programs. It is also important to note that, while the comparison is useful, there are limits to the strength of the conclusions that can be drawn, as different management arrangements were set up to respond to different operating contexts and to further different objectives. 3 Fund management arrangements

This chapter examines the Fund’s management arrangements and the extent to which they are appropriate, effective and efficient. It finds that the Fund’s management arrangements are appropriate and functioning well. Four issues where effectiveness and efficiency could be further improved are highlighted, while the Fund’s management costs are found to be relatively high. Moving forward, the evaluation finds the management arrangements will continue to be relevant for a future Fund phase.

3.1 Fund management arrangements are appropriate The evaluation finds that the Fund’s management arrangements are appropriate; that is, each element of the Fund management arrangements is needed to support CSOs to implement high quality WASH projects. When examining DFAT-funded CSO programs, including the CS WASH Fund, AACES and ANCP it was clear that they all include, or aspire to include, a number of common management elements: Some form of DFAT oversight, generally as part of a committee involving other key stakeholders Resources to manage programs on a day to day basis. Such resources are either in-house DFAT staff or externally contracted facilities Mechanisms to encourage NGOs to share knowledge and lessons from their activities Resources and systems to monitor and evaluate the program as a whole (not just individual CSO activities). The different programs have implemented these elements in different ways, as discussed in section 3.5 below. However, their commonality indicates some agreement in DFAT on the important elements that should be included in a CSO program. The completion review of the Fund’s first phase highlighted elements that would improve or contribute to the appropriate functioning of the Fund’s management arrangements. For example, the Review recommended that the current Fund include a specific learning component, and that an independent monitoring and evaluation panel be retained. The Fund includes all the program management elements which are common to DFAT-funded CSO programs and/or which were recommended in the first phase’s completion review. For example: DFAT WSH is closely engaged in oversight and management of the Fund, through participation in regular meetings with other members of the Fund management team The FMF handles much of the day to day Fund management work which DFAT WSH is not sufficiently resourced to undertake The Fund clearly focuses on, and has committed resources to, implementing a Knowledge and Learning component the MERP and the Fund’s M&E arrangements ensure CSOs have access to M&E expertise, and there are processes which monitor and report on Fund-wide achievements. The Fund’s program management arrangements therefore reflect broad DFAT views on important elements for managing CSO programs, as well as the findings of the evaluation of the Fund’s first phase. The evaluation team therefore judges that they are appropriate.

3.2 Fund management arrangements are functioning well The evaluation finds that the Fund’s management arrangements are, broadly speaking, functioning well. This conclusion was drawn by examining the MERP’s mid-term assessments of the Fund’s progress, as well as stakeholder views on Fund management. One of the key markers of the success of the Fund’s management arrangements is the effectiveness of CSO projects and the Fund more broadly. This assumes that if CSOs have been well supported by the Fund’s management arrangements, their projects are more likely to be effective and to contribute to the Fund’s success. This evaluation has not examined the effectiveness of CSO projects. However, it has examined the mid-term review documents developed by the MERP. It is clear from these documents that the Fund is broadly on track. The MERP notes that “Overall…adequate progress was made during the first two years of implementation” (M&E Note 5). Progress has been particularly strong in areas such as work on the enabling environment, gender equality and knowledge and learning (M&E Note 5). The effectiveness of CSO projects provides credible evidence that the Fund’s management arrangements are functioning well. Interviews with Fund stakeholders also demonstrated that the Fund’s management arrangements are largely functioning effectively and efficiently. When asked about the Fund’s strengths, all the main elements of the management arrangements (DFAT WSH, the MERP, and the FMF including the Knowledge and Learning component) were named at least once. Perhaps more importantly, many interviewees reflected that these different elements work well together. The Fund management team noted that all Fund management team members work in a collaborative, cooperative manner, and that the Fund’s management arrangements work well because of the quality and commitment of the team members involved. This view was largely reflected by CSOs, with a number noting that different Fund management elements act in an integrated manner, deliver consistent and useful messages, and are made up of high quality individuals. Two CSOs presented an alternative view, stating that the management arrangements involved too many layers and that it was challenging for CSOs to deal with the multiple groups involved. This was a minority view and, overall, the evaluation team finds that the Fund’s management arrangements are functioning well. 3.3 Strengths of Fund management arrangements Within the good performance of the Fund’s management arrangements, two areas were identified as particular strengths: the Fund’s Knowledge and Learning component, and CSO engagement with partner governments. Many aspects of the Fund’s performance assessment arrangements are also performing well, such as the MERP model, the flexibility provided to CSOs to adjust their projects over time, and the Fund’s approach to gender equality and disability inclusiveness. These are discussed in detail below.

3.3.1 Knowledge and Learning component A focus on knowledge and learning is one of the key features of the Fund. It takes a particular approach to knowledge and learning, having a specific Fund component (known as the K&L component) which is overseen by the Knowledge and Learning Manager (KALM). Overall, the evaluation finds that the Fund’s approach to K&L is one of its strengths. It provides a strong model which other programs in DFAT should learn from and could consider adopting and adapting to meet their particular needs. CSOs were very positive about the Fund’s focus on K&L. They reflected that other donor programs claim that knowledge and learning is important, but they rarely back this up with specific resources or activities. CSOs are therefore highly appreciative of the Fund’s focus on knowledge and learning and the corresponding resources dedicated to it, with some CSOs describing DFAT as a leading donor when it comes to knowledge and learning. The evaluation found that the K&L component is assisting CSOs to improve their projects. In interviews, CSOs reflected that the K&L component’s earlier focus on the Fund website and learning through electronic means had not been as helpful as desired. However, in recent years the K&L component has shifted to focus on face to face learning, particularly global and regional learning events, and these are proving very effective for CSO knowledge and learning. CSOs outlined that some of the benefits of learning events include: Increased knowledge and practices: 15 of 16 survey respondents stated that learning events had improved their project team’s WASH knowledge and practices either ‘significantly’ or ‘extremely’ (see Figure 2) Engagement with project partners: Partners, who are known as ‘change agents’ and are often from partner governments, have the opportunity to participate in learning events. This was seen by CSOs as highly beneficial as it exposed project partners to new ideas and practices Increased motivation: In-country teams, who can be quite isolated, are able to connect with the global WASH community and this can be very motivating.

Figure 2: Learning Events have improved WASH knowledge and practice

The Fund’s K&L component has also supported CSOs to improve their focus on gender equality and disability inclusiveness. The majority of survey respondents said that gender equality and disability inclusiveness were discussed often or frequently in K&L activities. 16 of 17 survey respondents also agreed or strongly agreed that the K&L component had assisted their project improve its approach to gender inclusiveness, while 14 of 17 survey respondents agreed or strongly agreed that it had assisted their approach to gender equality. While CSOs were positive about the Fund’s approach to K&L, they provided some suggestions for how it could be further improved. These are outlined below and should be considered for the remainder of the Fund’s current phase and/or its next phase: Drawing on experiences across CSO projects, the Fund could produce manuals or ‘how to’ guidance notes with practical information to assist project implementation Rather than K&L budgets being linked to specific projects, regional K&L budgets could be considered. This could, for example, be used to facilitate learning within a CSO that has projects in a number of countries Additional ‘collaboration’ funding could be made available to support collaboration between CSOs when such opportunities arise during project implementation.

3.3.2 Engagement with partner governments The Fund aims to actively engage partner governments at national and sub-national levels in the countries where projects are implemented, predominately through CSOs. This engagement between CSOs and partner governments appears to be one of the Fund’s strengths. The Fund strongly encourages CSO’s to work with ‘change agents’, many of whom are government counterparts. In interview and survey responses, CSOs reflected that they have close links with partner governments. They work with partner governments on project specific issues and, where possible, share project M&E data with them. CSOs also engage in broader WASH policy dialogue with partner governments, for example through existing WASH forums. The DFAT posts interviewed as part of the evaluation also felt that CSOs had good links with partner governments. Although the evaluation did not have the opportunity to verify these views with partner governments, it appears CSO engagement with partner governments is an appropriate and effective element of the Fund’s management arrangements.

3.3.3 Positive aspects of the Fund’s performance assessment arrangements The evaluation finds that many aspects of the Fund’s monitoring and evaluation arrangements are functioning well. The MERP model, the MERP support to CSOs and the Fund flexibility discussed below are considered strengths of the Fund’s management arrangements and demonstrate good practice which should be considered by other DFAT programs. A particular highlight is the M&E approach to gender equality and disability inclusiveness (discussed below) which, when combined with the gender equality and disability inclusiveness K&L activities (discussed above), provides a useful model which could be adopted by other DFAT programs. Many aspects of the Fund’s monitoring and evaluation arrangements are viewed positively by CSOs and the Fund management team. For example, the MERP model of having M&E experts engaged for the life of the Fund is seen as one of the Fund’s strengths. In the survey, 17 of 20 respondents agreed or strongly agreed that the MERP model was useful and helped improve their project. In interviews both CSOs and members of the Fund management team reflected that the model brings significant benefits. For example, it has ensured continuity and allowed MERP members to build deep knowledge of the Fund, its projects and the context in which they work. It is generally perceived that alternative models, such as contracting different M&E consultants for short-term work, would be less effective and efficient. One of the MERP’s main roles is to support CSOs to implement high quality WASH projects. Much of the support provided by the MERP is viewed very positively by CSOs. Generally speaking, the monitoring visits which the MERP has conducted are highly regarded. All survey respondents agreed or strongly agreed that monitoring visits helped improve their projects, and also agreed that further monitoring visits would be useful. In interviews, CSOs stated that monitoring visits have been valuable, providing critical questioning, advice and coaching which has assisted with project improvement. Monitoring visits are also valued by the Fund management team as they provide valuable information on in-country work. The MERP is also responsible for developing and maintaining the Fund’s performance assessment arrangements. Two aspects of these arrangements are viewed as highly beneficial. The first is that the Fund’s arrangements provide flexibility for CSOs to adapt and adjust their projects during implementation: in six monthly and annual reports, CSOs are able to suggest and have changes to their projects approved. Interviewees reflected that this is highly valued, particularly when compared to the practices of some other donors who provide much less flexibility. The ability to adjust projects is also valued given its consistency with the Fund’s focus on knowledge and learning. The approach to gender equality and disability inclusiveness in the Fund’s M&E system is also strong. When setting up the Fund’s performance assessment arrangements, the MERP included a specific outcome on gender equality as well as several key performance indicators (KPIs) and key performance questions (KPQs) on gender equality and disability inclusiveness. CSO Operational Plans (or project designs) also include a specific section on gender equality and social inclusion. This focus is supported during project implementation. For example, CSOs must report against the KPIs and KPQs, and gender equality and disability inclusiveness are regularly discussed in monitoring visits and K&L events. Fund reporting also includes a focus on gender and disability: the MERP’s mid-term review notes include a specific paper on gender and social inclusion, while these issues are integrated throughout the other MERP mid-term review notes. CSOs outlined that this focus has been very helpful, as shown in Figure 3. In interviews, CSOs noted that the Fund’s arrangements have ensured gender equality and disability inclusiveness are high priorities for project teams, with a number of CSOs reflecting that it pushed teams to go further in these areas than they might otherwise have done. Consistent with the findings of the MERP’s mid-term review notes, CSOs also reflected that their engagement with gender or disability-specific implementation partners has driven strong approaches to gender equality and social inclusion.

Figure 3: The Fund’s Performance Assessment Arrangements have improved approaches to gender and disability

3.4 Improving the Fund’s effectiveness and efficiency Although the Fund’s management arrangements are largely functioning well, four concerns about management effectiveness or efficiency were consistently raised during the evaluation. These are: The efficiency of CSO grant management The effectiveness of the Fund’s reporting arrangements The independence of the MERP, and Fund engagement with DFAT country programs. These are discussed below.

3.4.1 CSO grant management could be more efficient In addition to providing strategic oversight for the Fund, DFAT WSH holds the grant agreements with the 13 Fund CSOs. This allows DFAT WSH and CSOs to maintain a direct relationship. It also creates significant administrative work for DFAT WSH, such as maintaining grant agreements, processing payments and completing partner performance assessments. As noted above, DFAT WSH has contracted the FMF to undertake much of the day to day Fund management. This includes regular liaison with and mentoring for CSOs on budgets, financial acquittals and reporting. CSOs and the FMF will work together on particular issues and, where necessary, refer them to DFAT WSH for decision. Both DFAT WSH and the FMF questioned the efficiency of this arrangement. DFAT WSH has outsourced the day-to-day management of the grant agreements, yet it still holds the agreements and the risk associated with them. As a result DFAT WSH maintains close oversight of the Fund and project activities. This oversight means that the FMF has to work very closely with DFAT WSH, referring issues and seeking guidance on decisions it is not able to make. Interviewees noted that this can create double-handling and additional work for all parties involved. A more efficient mechanism for the Fund’s next phase may be for a future FMF7 to hold the grant agreements with CSOs. This would have the significant advantage of reducing DFAT WSH’s contract management work. The time saved could be used for other tasks where DFAT WSH can add significant value for the Fund, such as strategic oversight (including for M&E), building relationships with country programs and maximising public diplomacy opportunities. By increasing the scope of a future FMF’s work, economies of scale and an increase in facility efficiency may also be achieved. Such an approach also has disadvantages and risks, as summarised in Table 4. One risk is that the relationship between CSOs and DFAT WSH could potentially be reduced. When questioned, CSO interviewees were generally not overly concerned about this. The current system results in limited engagement between DFAT WSH and CSOs, and CSOs generally see this as acceptable. Many Australia-based CSOs noted they have other avenues to engage with DFAT, such as the WASH Reference Group8. Some CSOs expressed a preference for direct contracts with DFAT WSH, while others noted that the current system is working well for them and, from their perspective, changes are not needed. At the same time, there was a general sense that an alternative system could be feasible as long as lines of communication and engagement were clear. That said, a number of CSOs expressed a preference for some direct relationship with DFAT WSH, regardless of the contracting arrangements. CSOs who have had recent direct contact with DFAT WSH (for example, on project-specific challenges) stated that this was very useful as it provided an opportunity to directly engage with DFAT WSH on its policies and priorities. It created confidence in CSOs that they understood and could meet DFAT WSH’s needs. A further risk of a future FMF holding CSO grant agreements relates to the potential relationship between CSOs and a contracted Fund manager. One Australia-based CSO who has been accredited under ANCP felt that the Fund subjects them to unnecessarily “forensic” management, given their accredited status. Such close management is likely to continue if a future FMF were to hold CSO grant agreements. Given the current Fund includes (and future Fund is likely to include) non-accredited CSOs, it would be overly

7 Note it is not assumed a future Fund Management Facility would be managed by a for-profit managing contractor. A future FMF could be managed by a for-profit or not-for-profit entity. 8 The WASH Reference Group is a community of practice for CSOs and research institutions involved in WASH aid programs. DFAT WSH actively engages with the Reference Group. burdensome to have separate management approaches for accredited and non-accredited CSOs. This suggests it is necessary for DFAT WSH to clarify with CSOs that more stringent oversight will be implemented when compared to other DFAT funded CSO programs. The need to maintain a relationship between CSOs and DFAT WSH, combined with risks around DFAT ‘branding’ of the Fund (see Table 4), the lack of reporting feedback loops (see section 3.4.2), and management of accredited Australian CSOs suggests it may be useful for the Fund’s next phase to include a forum for DFAT WSH, other members of the Fund Management Team, and CSOs to meet on a regular basis (see section 3.6).

Table 4: What if a future FMF held funding agreements with CSOs? Disadvantages and risks Discussion and risk mitigation Greater contract costs for DFAT WSH due to Although there may be greater contract costs, DFAT WSH is increased contractor management fees likely to achieve better value by outsourcing grant management work and focussing on areas where it can add more value By increasing the scope of a future FMF’s work, a facility’s efficiency may increase Risk of reduced relationship between DFAT WSH A Fund advisory/consultative group which includes CSOs and and CSOs meets on a regular basis could be used to manage this risk Risk that accredited Australian CSOs feel they are A Fund advisory/consultative group which includes CSOs and being ‘overmanaged’ meets on a regular basis could be used to manage this risk by providing a forum to discuss management issues Loss of DFAT ‘brand’ for the Fund which may A Fund advisory/consultative group which includes CSOs and become more associated with a future FMF than with meets on a regular basis could be used to manage this risk, as DFAT could a greater focus by DFAT WSH on public diplomacy activities

3.4.2 Fund reporting arrangements could be improved As outlined in section 3.3.3, many aspects of the Fund’s performance assessment arrangements are functioning well. However, the evaluation also finds that the Fund’s project- specific and aggregate reporting could be improved, as discussed below.

CSO project specific reporting CSO views on the Fund’s reporting arrangements are mixed. CSOs complete six monthly and annual reports using the Fund’s Project Reporting Tool (PRT). The primary purpose of these reports is to fulfil accountability requirements by providing DFAT WSH with information on project progress. For this reason the PRT is relatively rigid, asking for specific data in a specific format to ensure it can be aggregated for Fund-wide reporting. Six CSOs reflected that the data that has to be collected and reported is appropriate for the size of the grants. Some of these have a view that the six monthly and annual reports are useful for their projects, for example by helping detect emerging issues early. This view was strongly reflected in the survey results, with 19 of 20 respondents agreeing or strongly agreeing that M&E reports are useful for their organisation. A further three CSOs had a clear understanding of the trade-offs involved in project reporting. They understood that CSO reports need to feed into higher level Fund-wide reporting, thus meaning the PRT needs to be relatively rigid. Such CSOs note this relatively rigidity is a disadvantage of being part of a bigger Fund, but it is outweighed by the advantages of being in the Fund such as access to the K&L component. Four CSOs were less positive about project-specific reporting. They expressed concern about the PRT, stating that it is too focused on compliance and ‘box-ticking’. They also feel it does not allow them to record the long-term story of change in their project, capture insights or learning, or facilitate a higher level of reflection on project performance. It is important to note that the PRT was never intended to assist reflection or learning; as noted above, it was designed as an accountability tool. The evaluation team views this focus on accountability as appropriate, as it is necessary for the Fund to have a reporting system which enables results to be aggregated and reported at the Fund-wide level. The evaluation team also concludes that some CSOs’ negative views of the PRT are a result of a mismatch between the PRT’s intended purpose and CSOs’ preferences for the purpose of reporting. The MERP is planning its own review of the PRT as part of the next round of in-country monitoring visits. This review will be used to inform PRT design for the Fund’s next phase and will able to consider issues with the PRT in greater depth than has been possible in this evaluation. The evaluation team strongly supports this review and suggests that it should actively consider whether the PRT could be amended so it not only enables accountability and Fund-wide results reporting, but also facilitates CSO learning and reflection. The MERP’s review could consider the PRT itself and the process used to draft it. For example, one CSO suggested to the evaluation team that CSO projects might be better supported if there were regular performance discussions between CSOs and the MERP. Such discussions could be incorporated into the PRT process and would be consistent with the Fund’s ongoing focus on knowledge and learning.

Fund-wide aggregate reporting CSOs and the Fund management team expressed concerns about the aggregate reporting that reflects on Fund-wide progress, achievements and evidence. Some CSOs feel that one of the MERP’s key value-adds is providing fund-wide aggregate reporting, outlining what the Fund has achieved and evidence-based lessons on what is and isn’t effective in the Fund. However, CSOs felt that such Fund-wide reporting had not been available to date. Instead, reporting is seen as uni-directional; CSOs provide project-level data but there is no effective feedback loop to help them understand the evidence emerging across the Fund. The MERP’s mid-term review documents may help close this loop (note they were being finalised as this evaluation was undertaken and the majority were not available to CSOs when they were interviewed). However, the evaluation team’s assessment of the MERP mid-term review documents suggests CSOs expectations are unlikely to be met. The documents (some of which are still drafts) are largely descriptive and they note that mid-term is too early in the Fund to draw conclusions on the effectiveness of different approaches to WASH. This is reasonable, however, it suggests two areas for further action: For the current Fund, end-of-Fund reporting will need to draw conclusions on and compare the effectiveness of different WASH work supported by the Fund (note the MERP’s mid-term review documents also state this) For future phases, ‘feedback loops’ should be created so that CSOs have better access to emerging Fund-wide lessons and evidence. Such feedback loops could also be used to manage CSO expectations on when Fund-wide evidence can reasonably be reported. As discussed in section 3.6, this could potentially be achieved through a Fund forum in which DFAT WSH, other members of the Fund Management Team, and CSOs to meet on a regular basis. DFAT WSH interviewees also expressed concern about the effectiveness of Fund-wide reporting. There was a sense that the Fund is collecting a lot of data but there is insufficient consideration of whether this data is useful and/or being used. DFAT WSH also feel that some data collection is not delivering good value as it is time and resource- intensive to collect while the benefits are not immediately apparent. ‘Approach mapping’ was cited as a particular example in this regard. In addition, DFAT WSH does not feel confident that Fund-wide reporting, such as annual synthesis reports and the MERP mid-term review notes, are meeting its needs. The evaluation team identified two factors which are contributing to this situation. First, there appears to be tension between the M&E data and analysis that the MERP would ideally like to undertake, and what DFAT WSH views as reasonable and appropriate to undertake. In interviews, DFAT WSH noted that the MERP’s independence needs to be tempered by strategic oversight to ensure their work remains within reasonable parameters. Second, DFAT WSH has found it challenging to define what they require from the Fund’s performance system. One interviewee noted that the DFAT team is often focussed on detailed management and has few resources to look at big picture, strategic issues. It is the evaluation team’s view that these issues have constrained DFAT WSH’s ability to provide solid strategic oversight to the MERP and ensure its information needs are met. This suggests that, for the next phase of the Fund, DFAT WSH needs to ensure it has sufficient time and resources to engage with strategic level M&E issues. It also suggests that DFAT WSH and the MERP need to create periodic opportunities to assess (or re-assess) the M&E and reporting information that is required and ensure these needs are being met. This may become more feasible in the Fund’s second phase, particularly if alternative contracting arrangements for CSOs are put in place.

3.4.3 MERP independence could be reconsidered Currently, the MERP is contracted directly to DFAT WSH and so is considered independent from the FMF and CSOs. This brings benefits, particularly for DFAT WSH. Because of the MERP’s independence, DFAT WSH can feel assured the data provided by the MERP is credible and that it hasn’t been filtered or made more positive by a managing contractor who has a commercial imperative to ‘sell’ the Fund’s achievements. The benefits of the MERP’s independence have not been as strong as anticipated. Early in the evaluation, members of the Fund management team stated that CSOs were likely to regard the MERP’s independence from the FMF as valuable. However, in interviews the majority of CSOs were ambivalent about the MERP’s independence and generally felt it had few direct benefits for them. Note that three CSOs were supportive of the MERP’s independence. During interviews a number of disadvantages and risks to the MERP’s independence were raised. These include: It creates greater administrative work for DFAT WSH, which must manage the three MERP contracts It creates risks for CSO projects because there is potential disconnect between project budgets and M&E (for example, the MERP’s work may point to project changes, but such changes may not be consistent with the Fund Manager’s views on appropriate use of budget) Without a formal contracting arrangement between the FMF and the MERP, the effectiveness of the management arrangements relies heavily on the goodwill and teamwork of the people involved. Although the relationships are currently working well, the contracting arrangements do not manage the risk that team relationships can break down or become dysfunctional. Because of these issues, alternative MERP contracting arrangements have been considered. One option is for all three MERP members to be contracted through a single contract with DFAT WSH. This would reduce DFAT WSH’s contract management load while retaining the MERP’s independence. Note that it would not address the other risks highlighted above. A more significant change that could be considered for the Fund’s next phase is contracting the MERP through a future FMF. The opportunities and risks associated with this are summarised in Table 5. One possible opportunity is that greater links between the Fund’s M&E and K&L component could be built. A number of interviewees suggested that the Fund’s M&E system and K&L component could be more closely linked. It is clear that the MERP and the KALM currently work together closely, that information from the M&E system is informing the K&L component, and that no significant disconnects between Fund M&E and K&L have arisen as a result of the current contracting arrangements. However, there is a sense that the links between the two areas continue to evolve and that greater synergies could be possible in the future. Such synergies may be more easily realised if the two components were brought under a single management structure. A further advantage could be that, by bringing Fund management team members under one roof, communication and team management could be streamlined. In particular, an overall team leader could allocate resources more flexibly to meet priority work demands. In interviews, the FMF highlighted that their resources have been stretched at peak work periods, such as just prior to learning events, and that greater flexibility in allocating staff resources would be beneficial. A significant risk of contracting the MERP through an FMF may be the availability of appropriate staff. In interviews, both CSOs and members of the Fund management team noted it may be difficult to attract MERP members of a similar high calibre if they are contracted through a future FMF, particularly if the FMF contract is won by a large managing contractor. The assumption is that only a small pool of skilled people who can fulfil MERP roles are available, and because of the demand for their time such people generally work in small or single-consultant companies. In summary, the MERP contracting arrangements for the Fund’s next phase should be carefully considered. Each of the possible approaches (retaining the current model, contracting the MERP to DFAT WSH under a single contract, or contracting the MERP under an FMF) has advantages and disadvantages. A key point is that the Fund’s management arrangements work because of the quality and commitment of the team members involved. It is the evaluation team’s judgement that, for the future phase, a key driver should be ensuring the best possible team is in place to manage the Fund. This should be the key factor informing decisions about the contracting model for the MERP, rather than questions of independence.

Table 5: Opportunities and risks of contracting the MERP through a future FMF Opportunity Risk Fund effectiveness Reduced risk of disconnect between M&E and budget (in Loss of MERP independence; DFAT may have less relation to CSO projects) confidence in credibility of M&E data Greater links between K&L and M&E could potentially be It may be difficult to attract MERP members of similar realised high quality if they are contracted through a future FMF Fund efficiency Fewer contracts for DFAT WSH to manage Greater contract costs for DFAT WSH through increased management fees for a future FMF. Note this could be offset if MERP members were moved from daily to monthly rates A single team within an FMF could result in streamlined communication and management of resources

3.4.4 Engagement with DFAT country programs could be improved When the Fund was being designed it was clear that interest from DFAT country programs was variable. DFAT Posts were given the option to ‘opt-in’ to the Fund, or to be more actively involved in CSO selection and oversight. However, if Posts were not able to or did not wish to engage there was no obligation for them to do so. Regardless of their ‘opt-in’ status, DFAT WSH provides Posts with annual summary reports of Fund projects in their countries. The evaluation team interviewed five DFAT Posts to seek their views on the Fund. CSOs and the Fund management team were also asked about their engagement with DFAT country programs. These interactions demonstrate that engagement with the Fund by Posts continues to be highly variable. Some Posts are actively involved, convening CSO partners working in their countries, participating in MERP monitoring visits and engaging with learning events. Other Posts have rarely engaged, largely due to resourcing pressures and challenges prioritising the Fund. Some staff, particularly A-based staff in Posts with recent turnover, were not aware (or only peripherally aware) that the Fund was supporting CSOs in their country. This variable engagement is consistent with the experience of other centrally managed DFAT programs such as ANCP (see ODE evaluation of ANCP, 2015). While engagement between some Posts and the Fund has not been ideal, this does not appear to have created major issues. Rather, the low level of engagement is seen as a missed opportunity. All groups interviewed indicated that greater engagement between the Fund and country programs is desirable. DFAT WSH reflected that there could be a number of benefits of greater engagement, for example, Posts could play a greater role in coordinating and/or overseeing CSO work. Posts and DFAT WSH could also work together to identify increased public diplomacy opportunities and improve the links between the Fund and country programs, for the benefit of both parties. A number of CSOs noted they had attempted to engage Posts but had struggled to gain traction. 64 per cent of survey respondents indicated they had contact with posts at least once or twice a year and 60 per cent of survey respondents indicated they felt their engagement with Posts was ‘about right’. Note Posts indicated they were grateful when CSOs proactively engaged, for example through face to face meetings and invitations to participate in public events. While Posts agreed that greater engagement between the Fund and Posts would be beneficial, the level of engagement will continue to need to be tailored to individual Post needs. For example: Some Posts indicated they would like to better understand Fund projects, for example through regular reporting and involvement with monitoring visits. This can help ensure links with bilateral WASH activities or, at the least, avoid duplication Some Posts continued to want strong engagement, for example, an understanding of the overarching lessons coming out of the Fund so that they could be applied to bilateral WASH programs Particularly active Posts indicated they could coordinate CSO projects in-country, for example to build greater links between CSO projects and the partner government. This would be appropriate where an existing in-country coordination mechanism did not exist All Posts interviewed indicated they would like to be involved in CSO selection for the next Phase. While all parties wish to engage more, broad DFAT experience demonstrates that staff at Post are busy and have competing priorities to manage. This makes it incumbent on the Fund management team to actively understand Posts’ needs and attempt to meet these needs efficiently. Some suggestions provided during interviews were: Posts are more likely to engage with DFAT WSH than with an FMF Desks in Canberra should be kept in the loop It could be useful for the Fund to check in with country programs twice a year to ensure that any new staff are aware of the Fund and have the information they require Discussions between DFAT WSH and Posts, rather than email, would be beneficial CSOs should continue to be encouraged to build relationships with Posts. These findings indicate that, to maximise engagement between the Fund and country programs, DFAT WSH prioritise engagement with country programs. The focus of such engagement could be increased public diplomacy and publicity for the Fund as well as greater links between Fund and bilateral activities, given these two areas are mutually beneficial to DFAT WSH and country programs. Higher prioritisation of this work may be difficult for the remainder of the Fund’s current phase but may become feasible for the Fund’s next phase, particularly if the management of CSO grant agreements is adjusted (see section 3.4.1).

3.5 Fund management costs are relatively high but are justified As discussed in the previous sections, the Fund’s management arrangements are generally operating well to support CSOs and Fund-wide success. However, given the resources devoted to Fund management it can be viewed as a relatively high cost model. This section addresses the question of whether the costs are justified by comparing the Fund to two other DFAT-funded CSO programs: the Australia Africa Community Engagement Scheme (AACES) and the Australian NGO Cooperation Program (ANCP) (see section 2.3.1 for background on these programs). The comparison draws heavily on recent independent evaluations of AACES and ANCP. Consistent with the focus of this evaluation, the comparison focusses on the effectiveness of each program’s management arrangements, rather than the effectiveness of CSO activities they fund. When examining management costs, the evaluation considered all costs associated with the common management elements for CSO programs outlined in section 3.1 – that is, DFAT oversight, day to day program management, mechanisms to share knowledge between CSOs, and program M&E. In summary, the evaluation finds that although the Fund’s costs are higher than other programs, they are justified because of the effectiveness of the Fund’s management arrangements. For future Fund phases, the Fund would benefit from systems which can be used to justify its relatively high management costs. A summary of the effectiveness of each program’s management arrangements, as well as their program management costs, is at Table 6. The table highlights the following key points. The ANCP has low management costs, with less than 2 per cent of program value being used for program management. Eight DFAT full-time equivalent (FTE) staff manage the program, a relatively low commitment given the size of the program. In a given year, ANCP expenditure was around eight times that of AACES but had only three times as many staff. At the same time, the ANCP evaluation highlighted some issues with the program’s management arrangements. At the time of the evaluation ANCP did not have a theory of change or a system for collecting and reporting on high level outcomes. Cooperation between NGOs and learning from the monitoring and evaluation system was also limited. In comparison, the effectiveness of the management arrangements for both AACES and the Fund are similar, and relative high. Both Funds have clear theories of change and have strongly supported CSOs to implement gender equality and disability inclusive activities. Both have well-regarded approaches to knowledge and learning. One difference is around program-wide performance assessment arrangements. The Fund has clear Fund-wide performance assessment arrangements which can be used to describe the Fund’s achievements (although note the current issues with this described above). Given AACES supported three different sectors, it struggled to build and describe a coherent program- wide story of achievements. On first glance, the program management costs for the Fund appear significantly higher than for AACES. To a certain extent this reflects the different approaches to monitoring and evaluation and knowledge and learning that the two programs have taken. The Fund has contracted expertise and support in these areas, and as a result, a relatively high 7.9 per cent of Fund value is committed to program management contracts. Some specific elements of the Fund’s approach, for example allowing CSOs flexibility to adapt their projects (see section 3.3.3) are also resource intensive. In comparison, AACES received some support from a Resource Facility and around 4 per cent of program value was dedicated to program management contracts. AACES also relied heavily on CSOs to provide monitoring and evaluation and knowledge and learning support to each other. For example, CSOs organised and convened learning events on a rotating basis. They also peer reviewed each others’ work, including by conducting field visits. This would have taken a significant time commitment from CSOs, a commitment which has not been quantified and is hidden when looking at the program management contract costs. Even taking into account the CSO time commitment required to implement AACES, the Fund’s costs are relatively high. The Fund’s current investment in program management costs still appears reasonable, particularly as they are lower than what was planned in the Fund’s design. In the evaluation team’s judgement it is also a good investment given the performance of the management arrangements. For a future phase it is possible that the Fund’s management costs will increase further, particularly if alternative CSO and MERP contract management arrangements discussed above are implemented. This suggests two areas of consideration for the current and future Fund phases: For the next phase, the Fund could consider adopting some elements of the ‘partnership’ approach used by AACES. For example, the Fund could reduce the cost of contracted expertise by having CSOs peer review each other’s work (including by conducting monitoring visits to each other’s field sites), or by having CSOs convene learning events on a rotating basis. But while this may reduce the contracted Fund management costs, it is unlikely to decrease the overall costs of the Fund because of the additional time commitment needed to implement a partnership approach. Some elements of the AACES partnership approach may also be more difficult for the Fund to implement, given it operates across several regions and supports CSOs based outside Australia For the Fund’s next phase, the Fund should consider how it can justify its relatively high management costs. Ideally the Fund would attempt to demonstrate that its investment in management (including for M&E and K&L) has contributed to more effective CSO projects. It is noted that this may be practically and methodologically challenging, and that a more feasible approach may be ensuring the future Fund’s M&E system monitors and reports on the effectiveness of the management arrangements. An independent review of the future Fund’s management arrangements, similar to this evaluation, could also be considered.

Recommendation 1

For the Fund’s next phase, the Fund’s monitoring and evaluation system includes arrangements which ensure the Fund’s management costs can be justified. Table 6: Comparison of program management effectiveness and program management costs Management Arrangements CS WASH Fund ANCP9 (ODE evaluation, 2015) AACES (Completion review, 2016)

Summary of program 13 CSOs implementing 29 projects in 19 53 ANGOs implementing 575 projects in 57 10 ANGOs implementing projects in 11 African countries. Focussed on a single sector (WASH) countries. A broad range of sectors are covered countries. Focussed on three sectors (WASH, food including health, WASH, education, governance security, and maternal and child health) and agriculture Program oversight Oversight through quarterly management team Oversight by DFAT. The ANCP Committee for Oversight by a program steering committee in which meetings and bi-weekly teleconferences Development Cooperation oversees the DFAT and each NGO had equal representation and involving DFAT WSH, FMF and MERP accreditation process. Highly regarded by NGOs responsibility for program decision making. Highly regarded by NGOs Program management Management by the FMF, with oversight by Oversight and day to day management by DFAT Management by DFAT and NGOs with support from a DFAT WSH Resource Facility Theory of change Clear Fund-wide theory of change on how No program-wide theory of change Clear program-wide theory of change on how NGOs change can happen in the WASH sector can facilitate change. No sector-specific theories of change Program-level performance Clear and detailed Fund-wide performance No performance assessment framework for It is unclear from the Review whether AACES had a assessment assessment arrangements collection and analysis of high level outcomes program-wide performance assessment framework. The Review reports that, given the program supports three sectors, it has been challenging to build and present a coherent program Support for gender equality and Strongly supported through project design, Fund Strongly supported through ANCP guidelines. Strongly supported through program design and disability inclusiveness performance assessment arrangements and ANCP has influenced how NGOs address cross- support/workshops provided during the program reporting, and knowledge and learning cutting issues component Approach to Knowledge and Specific K&L component managed by a Some learning is facilitated between a small Focussed on peer to peer NGO learning. For example, Learning KALM. The approach is well-regarded by CSOs number of partner NGOs. However, the learning NGOs facilitated bi-annual learning workshops; peer potential of ANCP’s learning framework has not reviewed each others’ work; and conducted field visits been maximised and it is recognised learning to each others’ projects. The approach was highly could be better facilitated regarded by NGOs Management costs CS WASH Fund ANCP AACES Program value $103m over five years (on average, $20.6m per Approximately $130m per year $83m over five years (on average, $16.6m per year) year) DFAT FTE (from DFAT WSH and 1.5 FTE 8 FTE 2.5 FTE the ANCP evaluation)10 Management contract costs (from 7.9% of Fund value Less than 2% of program value 4.1% of program value Aidworks)

9 The information on ANCP relates to the when the ANCP evaluation was finalised in 2015. It is likely a number of issues have been addressed or improved since that time. 10 DFAT staff FTE has not been converted to a monetary value. The evaluation team did not have data on the seniority of staff involved in ANCP and AACES, and so could not compare the costs of DFAT staff across the programs. 3.6 Moving forward A key objective of this evaluation is to consider whether the Fund’s management arrangements are relevant to the changing context of the aid program. An understanding of this will help inform the management arrangements for the Fund’s next phase. The changing context of the aid program includes: Constrained resources: this includes constrained budgets and, more particularly, limited staff numbers Reduced aid management expertise: with the integration of DFAT and AusAID, a smaller proportion of DFAT staff have expertise and experience in aid management Public diplomacy: the integrated department has a high expectation that aid activities will provide public diplomacy opportunities Geographic priorities: the aid program has increased its focus on the Indo-Pacific and reduced focus on Africa. The particular context for the Fund has also changed since it was designed in 2012. At that time the Fund was approximately 10 per cent of the aid program’s WASH funding. By 2016-17 this has increased to 46 per cent of the aid program’s WASH funding, making the Fund a more prominent part of DFAT’s support to the WASH sector. In this context, it is the evaluation team’s judgement that the Fund’s approach to management will remain relevant for the Fund’s next phase. The key elements of the Fund’s management arrangements, including DFAT’s strategic oversight, day to day management by an FMF, the Knowledge and Learning component and the resources and effort devoted to Fund M&E, remain appropriate to the changing context and should be retained. With reduced staff resources, it is appropriate for DFAT WSH to focus its scarce resources on areas where it can add the greatest value: policy and strategic oversight (including oversight related to M&E), maintaining relationships with country programs, and public diplomacy. The need for DFAT WSH to focus on the latter two areas appears more important now than when the Fund was designed, given the Fund represents a greater proportion of the aid program’s WASH funding. In light of these factors, the evaluation team finds it is appropriate for CSO funding agreements to come under a future FMF, as discussed in section 3.4.1. It is also appropriate for DFAT WSH to contract expertise that is scarce in the department, in areas such as monitoring and evaluation and knowledge and learning.

Recommendation 2

That for the Fund’s next phase, DFAT WSH commits its resources to areas where it has the greatest value add, particularly strategic oversight (including for M&E and reporting), relationships with country programs and public diplomacy. At the same time as outsourcing Fund management and specific expertise, DFAT WSH should continue to have strong oversight of the Fund and maintain relationships with CSOs in the sector. This reflects the experience of the Fund’s current phase, as well as the experience of other DFAT funded CSO programs – in particular AACES, which found that partnerships between DFAT and CSOs maximised the program’s results. To a certain extent DFAT WSH can maintain its relationship with CSOs through ongoing engagement with the WASH Reference Group. For the Fund’s next phase, DFAT WSH should also consider a Fundspecific consultative or advisory group that includes DFAT WSH, the FMF, the MERP and CSOs, as discussed in sections 3.4.1 and 3.4.2. This group could meet on a regular basis (possibly six-monthly) and could provide advice to inform Fund decision making. It could also address a number of issues highlighted in this evaluation, for example: It could ensure DFAT WSH and Fund CSOs continue to have a direct relationship. DFAT WSH would be able to regularly share its priorities and CSOs could feel assured they understand DFAT WSH’s needs and are able to meet them It could help ensure the Fund continues to be strongly branded as a DFAT program It could provide a forum where management issues, such as appropriate CSO and Fund-wide reporting, could be discussed and negotiated, and It could be used to create a reporting ‘feedback loop’ so that CSOs were aware of Fund-wide reporting and evidence. The exact remit of the group, as well as its intersection with other groups such as the WASH Reference Group and the Fund’s K&L advisory group, should be further explored during the design of the Fund’s next phase.

Recommendation 3

When designing the Fund’s next phase, the design team considers a Fund-specific advisory or consultative group which includes all members of the Fund management team and Fund CSOs. By acting in this and other areas highlighted in this chapter, DFAT WSH will ensure the management arrangements for the Fund’s next phase are effective, efficient, and relevant to the DFAT context. 4 CSO Selection

This chapter examines the extent to which the Fund’s initial selection process and inception phase enabled the selection of the most effective CSO projects. Understanding the advantages and disadvantages of how these processes were implemented can then inform the next Fund’s design.

As noted in section 2.4 on this evaluation’s limitations, the evaluation team does not judge whether the most effective CSOs and projects were selected. Analysis is solely of process and whether it could be improved.

4.1 Improving the Fund’s CSO selection process

4.1.1 The CSO selection process had many strong elements The CSO selection process displayed a number of strengths. Fund management team stakeholders (DFAT WSH, MERP and FMF) noted that some of the funded projects were clearly outstanding. Fund management stakeholders and CSOs noted the Fund’s design incorporated the findings from the independent completion review of the Fund’s previous phase in providing a framework for deciding which CSOs and projects to now support. The selection panel members interviewed appreciated the staged approach to CSO selection, the Selection Guidelines’ clearly weighted criteria, and that panel membership combined expertise from the WASH and NGO sectors.

4.1.2 Fund stakeholders perceived a disconnect between some projects and the Fund’s design intent and theory of change The Fund management team noted that some of the Fund’s CSO projects were outstanding but a minority of projects did not align with the Fund’s design intent. They were surprised by their selection. The Fund’s purpose was to support projects which ‘raise the bar in terms of [WASH] program quality and sustainability.’11 Some CSO program managers also shared the Fund management team’s view and were confused as to why some CSOs received funding while other CSOs which applied did not. Fund management team members said that approximately 20 per cent of projects (6 to 7 of 29) were focussed on direct delivery of WASH services, rather than focussed on influencing sustainable change in the WASH enabling environment (as intended in the Fund’s design document and theory of change). According to Fund management team stakeholders, the CSOs implementing projects with this service delivery focus tended to be either CSOs that were: Not as experienced in delivering projects in the WASH sector (although experienced in development); or Experienced in WASH but have adopted a direct service delivery approach. While agreeing that some CSO’s approaches did not seem to match the Fund’s design, a minority of CSO staff noted that the Fund’s ‘actor-centred’ theory of change also went significantly further than the Fund’s design intent. The Fund’s theory of change is reflected in the Fund’s performance assessment arrangements. It was not explicitly outlined until the Fund’s inception phase in 2013 when the Fund’s

11 Design document at p. 1. performance arrangements were developed. This means that CSOs did not have the full picture of the Fund’s theory of change when they applied for the Fund. An implication of this is that the design for the next phase of the Fund should clearly outline the Fund’s theory of change. The theory of how sustainable change occurs in the WASH sector should also be reflected in a threshold or key selection criterion that is integrated into the assessment of organisational capability and of individual project proposals (described further below).

4.1.3 Key selection criteria did not appear to be given sufficient weight, including CSO’s past experience The Fund’s Selection Panel Guidelines arguably did not give sufficient weight to selection criteria that supported the Fund’s key objective (raising the bar in terms of WASH program quality and sustainability). In hindsight, given issues with some funded projects, it appears that particular criteria could have been given greater weight or set as a minimum threshold to reduce the risk that projects focussed on direct service delivery would be selected. Figure 4 (below) describes the selection process, with separate assessments made of an applicant CSO’s organisational capacity and country-specific project proposal. Tables 7 and 8 provide the selection criteria ratings. Note that: A CSO’s WASH specific expertise represented only 20 per cent of the assessment of the CSO’s organisational capacity (see Table 7 – highlighted in red) The CSO’s theory of change to implement the project represented only 10 per cent of the separate and subsequent assessment of the project proposal (see Table 8 – highlighted in red) The project proposal score ultimately determined whether a CSO was funded under the Fund; that is, a CSO’s organisational capacity and project-specific scores were not added to give a cumulative total. The selection process did take into account that assessing likely WASH project sustainability is related to a number of assessment criteria, such as the capability to engage with local actors, among other criteria listed. However, the implication is that an application could achieve a competitive score overall despite potentially being weaker in a criteria which is considered key (that is, applying innovative WASH approaches rather than a service delivery model).

Recommendation 4

That for future CSO selection, consistency with the Fund’s intent/theory of change is a heavily weighted criteria. Figure 4: CSO Selection Process

Table 7: Selection criteria weightings – initial assessment of CSO organisational capacity Note this was a ‘pass-fail’ step in the assessment process. CSOs had to achieve a score of at least 60 per cent to proceed to the next stage of the assessment.

1. Understanding of WASH sector challenges and effective WASH program 20% approaches 2. Capacity to achieve sustainable WASH outcomes for women, the poor 15% and socially excluded groups 3. Demonstrated high quality implementation of key AusAID policies 10% 4. Capacity to manage and support the proposed activities and ensure high 30% quality planning, performance management and learning throughout the planning and implementation period 5. Capacity of key personnel, particularly the dedicated WASH Program 25% Manager

Table 8: Selection criteria weightings - Project-specific selection criteria If a CSO achieved the requisite 60% score for its organisational capacity, its project-specific proposals were assessed against the following criteria.

(A) In-Country Delivery Capacity (50%) 1. Understanding of the country context and WASH needs 15% 2. Capacity to engage with local and national government and the broader 10% WASH sector to influence WASH policy and approaches 3. Capacity to achieve sustainable WASH outcomes for women, the poor 10% and socially excluded groups 4. Capacity to manage and support the proposed activities and ensure high 10% quality planning, performance management and learning throughout the planning and implementation period 5. Capacity of key personnel 5% (B) Activity Concept (50%) 6. The theory of change and implementation strategies are feasible and build 10% on current knowledge of effective approaches for sustainable WASH outcomes 7. The concept design is relevant to the country context and will contribute 15% both to the achievement of Fund outcomes in that country and AusAID WASH strategies (where relevant) 8. Gender equality and social inclusion are an integral part of the concept 10% design 9. The capacity building approach is likely to influence the sustainability of 10% WASH outcomes 10. The activity will contribute to the knowledge base for the development of 5% sustainable WASH outcomes

Fund stakeholders also noted that the assessment process did not take into account past CSO performance and that this was an oversight. This aspect was noted in interviews and through the CSO survey. Within the current Fund, CSOs were required to provide a ‘Past Experience Form’ to support the assessment stage of individual project proposals. However, this requirement did not have the effect that otherwise may have been intended (coming later in the assessment process, and not being weighted in the Selection Panel Guidelines). CSO past experience within the WASH sector or within the previous Fund was therefore not an intrinsic part of assessing that CSO’s claims. Future CSO selection under the next Fund may therefore wish to ensure a CSO’s past performance within the WASH sector and within the current Fund (where applicable) is an essential part of assessing CSO’s claims. A selection panel should consider this past performance, such as Partner Performance Assessments for DFAT-funded programs, in accordance with relevant rules and guidelines.12

Recommendation 5

That for future CSO selection, CSO performance in the WASH sector and the current Fund is explicitly taken into account.

4.1.4 The selection panel’s diverse expertise could have been better drawn upon and managed Some Fund management team stakeholders raised that the Fund’s selection panel did not have the right expertise to select the Fund’s CSOs. The Fund’s selection panel comprised four members: two WASH

12 For example, the Commonwealth Grants Rules and Guidelines, July 2014. members (from DFAT WSH and an external consultant) and two members from DFAT’s NGO section. The evaluation team makes the following observations on the selection panel: It was difficult to identify and contract a suitable external WASH expert to sit on the selection panel. DFAT WSH contacted several different consultants until a person able and willing to undertake the task was identified. Interviewees from the Fund management team suggested this consultant’s expertise may not have been ideal for the selection process To assess CSO’s statements of organisational capacity, the selection panel split into two teams (each with a WASH and NGO representative). To assess the 115 country-level project proposals, the two WASH representatives formed one team (assessing a greater number of proposals) while the NGO representatives formed another team (assessing a smaller number of proposals). This meant that this stage of assessment did not draw on the potential strengths of having a diverse team The participation of the NGO representatives was highly valued by the DFAT WSH panel member. At the project level stage of assessment, the NGO representatives also selected proposals from CSOs which were considered by the Fund management team to be outstanding For certain project proposals there was a significant difference between WASH representatives’ scores - up to 17 marks. In reviewing a sample of five ‘outlier’ CSO projects (those projects described by the Fund management team as adopting a service-delivery approach and a proposal from a high-performing CSO from the previous Fund that was not successful in obtaining funding) the evaluation team found that the WASH representatives typically had a significant difference between their scores. The scores of the DFAT WSH Section representative typically were more in line with the later assessments of projects by the Fund management team. This suggests that the external consultant had different views on how to approach key selection criteria, including criteria on gender equality and social inclusion (see section 4.1.5). These findings suggest that future CSO selection processes could ensure that: Panel members have appropriate WASH and other expertise which enables them to appropriately assess the focus areas of the Fund Panel members share the same views of what the Fund is aiming to support. For example, a similar view of what reflects best practice and how change occurs in the WASH sector and for gender and social inclusion in WASH (see section 4.1.5) Teams are suitably diverse and consistent throughout the process. This will reduce the risk of inconsistent assessment of proposals, and Appropriate dispute resolution processes are in place in case there are significant differences between panel members’ scores. This seemed to be in place for the assessment of organisational capability stage, but could have been strengthened at the project proposal assessment stage. Differences of ratings of up to 20 marks were allowed under the selection process guidelines; in hindsight, given concerns over some of the projects selected, this difference seems too large.

4.1.5 Focus on gender and disability inclusion in CSO selection The focus on gender and disability inclusion in the Fund’s selection process appears appropriate. For example: The assessment of CSOs’ general capacity to achieve outcomes for women and socially excluded groups formed 15 per cent of the CSO’s organisational capacity score (see Table 7) A further 10 per cent was applied for implementation of key AusAID policies, including on gender equality and disability inclusiveness (see Table 7) For project-specific proposals, the criteria ‘capacity of achieve sustainable WASH outcomes for women, the poor and socially excluded groups’ and ‘gender equality and social inclusion are an integral part of the concept design’ were each given 10 per cent weightings (see Table 8). When it came to assessing project proposals, however, the evaluation team found there were sometimes significant differences between WASH representatives’ scores on the gender and social inclusion criteria (based on a random sample of proposals). This could result in the DFAT WSH representative rating a CSO’s response to these criteria as ‘poor’ and the consultant rating these same criteria as ‘good’, to provide one example. This suggests that for a future phase panel members should have sufficient expertise in gender equality and disability inclusiveness, as discussed in section 4.1.4 above.

Recommendation 6

That for future CSO selection, panel members have appropriate expertise (in WASH, gender equality and disability inclusiveness, and with CSOs). Gender equality and disability inclusiveness were also a strong focus in the Fund’s inception phase. In their Operational Plans, CSOs were required to include a specific gender equality and social inclusion Plan to ensure these issues were appropriately addressed. CSOs were also actively encouraged to enter into partnerships with organisations with specific expertise in gender equality or disability inclusiveness (see MERP M&E Note 9, Gender and Social Inclusion). ODE agrees with the option raised by some CSO stakeholders to improve gender equality in the next Fund by specifically including a focus on how to reorient the traditional roles of men in WASH activities. This would be in addition to supporting the increased participation of women in non-traditional roles.

4.1.6 DFAT WSH inadequately resourced for CSO selection All DFAT WSH stakeholders (former and current) stated that the selection panel was under-resourced and DFAT lacked appropriate corporate systems at the time of CSO selection. This caused a huge workload in particular for the DFAT WSH staff member on the selection panel, who had to manually upload grant applications onto the corporate drive. This suggests that for future selection processes, the DFAT WSH team should consider using online grants management systems and ensure they have sufficient staff resources to manage the process. Such resources are only likely to be required on a short term basis, but could deliver significant value to DFAT WSH by helping ensure appropriate CSO projects are selected.

4.1.7 Transparency An intention of the Fund was that CSO selection should be transparent – that is, the selection process should be clearly articulated to ensure CSOs understood the basis for funding decisions. This transparency aimed to remedy a criticism highlighted in the previous Fund’s independent progress report.13 The DFAT WSH panel member is to be commended for keeping detailed records of CSO selection, both of individual applications and comparative data. The DFAT WSH team was able to provide detailed feedback to CSOs on how their application fared relative to successful applications. Given some of the issues with respect to the CSO selection process (noted above) some CSOs may nevertheless have found some of the results of the CSO selection process inconsistent with the Fund’s stated purpose. This finding also appears borne out by the CSO survey results. The survey response was too small to draw clear trends, however 18 CSO staff responded who were involved in developing their CSO’s applications to the Fund. Eleven responded from unsuccessful CSOs and seven responded from successful CSOs. Most staff from unsuccessful CSOs agreed that they had clear information with which to develop submissions,14 but most could not understand how applications were assessed or funding decisions were made.15

13 Independent progress report for Civil Society Water, Sanitation and Hygiene Fund, February 2011, p. 5. 14 Seven out of 11 respondents agreed, with one agreeing strongly. 15 Only three out of nine respondents who answered this question agreed that they could understand the basis for assessment. Three out of nine also agreed that they could understand the basis for funding. Most staff from successful CSOs strongly agreed that they had clear information to develop submissions.16 Most staff from successful CSOs also agreed they knew how applications were assessed,17 but only a minority agreed that they understood how funding decisions were made.18 Other factors could also explain the increased numbers of respondents who could not understand the basis for funding decisions. Most in-country project managers and CSO staff generally would have access to written guidelines for project proposal development, but may not have access to final results unless these were requested from DFAT and passed on by CSOs’ headquarters. To increase transparency, the next Fund could potentially hold a group feedback session on CSO selection following the announcement of successful CSOs (in addition to the information session on the Fund which is usually held for CSOs after the call for proposals and before the proposal deadline). DFAT WSH could also continue to provide individualised feedback on the assessment of CSOs’ applications to CSOs that request it.

4.1.8 Other suggestions for the next Fund’s design: regional allocations and size of individual CSO projects Aside from other issues regarding CSO assessment, DFAT WSH may wish to consider the available pool of CSOs further when designing the next Fund in two respects: the CSOs operating in particular regions and the size of individual CSO projects. 44 CSOs were found to have the requisite organisational capacity to then have their project proposals reviewed and some CSOs submitted proposals in multiple geographic regions. A greater number of CSOs however submitted applications to implement activities in some geographic regions over others. The majority of proposals were for South East Asia and for Southern Africa (see Figure 5). Overall only eight eligible CSOs submitted proposals for the Pacific region. Three of these CSOs were funded, with the bulk of Pacific proposals managed by one CSO. In designing the next Fund, the lower number of eligible Pacific-based applications reviewed raises a potential question of how the Fund could support greater Pacific-based WASH capacity. This is particularly the case if the next Fund decides to change its geographic distribution (with lesser support for African-based CSOs, as an example). In designing the next Fund, DFAT WSH may therefore wish to further consider the available regional pool of CSOs, particularly in the Pacific, and what impacts this could have for Fund selection and management.

Figure 5: Eligible CSOs and regions for which they submitted project proposals

DFAT WSH may also wish to reconsider the expected project size likely to be managed by CSOs. The Fund supports a greater number of smaller-sized grants than that originally envisaged under the design. The Selection Panel Guidelines noted that grants for individual CSO projects were expected to be in the order of $7 - $10 million across four years. A minimum of $2 million was also provided. Of the 29 projects supported, only two are valued at $7 million or above. The remainder of the projects are less than $5 million, with one third of projects less than the expected grant minimum. As project grant size impacts on the number of projects funded and the resources allocated for Fund management, DFAT WSH may wish to investigate with CSO’s directly their general capacity to manage projects of a particular size.19

16 Six out of seven respondents agreed, with three agreeing strongly. 17 Six out of seven agreeing, with two agreeing strongly. 18 Three out of seven agreeing, with two agreeing strongly. 19 ‘Officials are encouraged to seek input from potential grant applicants when developing grant activities’: Department of Finance, Commonwealth Grants Rules and Guidelines (July 2014), at para 8.2. 4.2 General satisfaction with the Fund’s inception phase

4.2.1 Benefits of inception phase The inception phase had a number of benefits for CSOs and DFAT WSH which point to this Fund feature being retained. For CSOs, the inception phase lowered the cost of developing proposals as detailed project designs were not required upfront. During the 3 month inception phase, 70 per cent of the cost of project design refinement was supported by DFAT WSH with 30 per cent of the cost borne by CSOs. CSOs also benefitted from MERP support for the development of Operational Plans. For DFAT WSH, the inception phase managed risk. The MERP and the FMF made recommendations to DFAT WSH on whether to fund each CSO’s operational plan. CSOs were also required to have a number of plans within their overall operational plan, specifically: a WASH plan; gender and social inclusion plan; environment, climate change and disaster risk reduction plan; knowledge and learning plan; sustainability and exit plan; a risk assessment plan; and an M&E plan. The Fund management team (and CSOs) could then track how certain issues were being managed by an individual CSO, and by the Fund as a whole. The inception design workshop held in Canberra in April 2013 was almost universally highly praised by CSO staff that attended. Of the 17 CSO staff interviewed, seven participated in the inception phase workshop and noted it as a Fund highlight. All four CSO project managers who responded to the survey on the inception phase workshop said it was ‘extremely’ or ‘significantly’ beneficial in helping CSOs improve their Operational Plans.

4.2.2 Inception phase length Fund stakeholders differed on whether the three month inception phase was a sufficient timeframe. DFAT WSH and some MERP members believed that a 6–12 month timeframe was preferable. Some CSOs have requested frequent changes to the implementation of their projects. A longer inception period may have given these CSOs more time to understand the relevant context and to anticipate appropriate responses. CSOs almost universally stated that a three month inception phase was appropriate. With the current Fund, for some CSOs there was a significant delay from the time of submissions (2012) to actual program implementation (2014) caused by the Australian government election (2013) and associated changes to the aid program, which caused delays for the Fund.20 CSOs reflected that this delayed start caused some problems (costs had increased with the fall of the Australian dollar, CSO staff had moved on, and original plans had changed as governments in partner countries ‘reallocated targeted areas etc’). It is difficult for the evaluation team to gauge whether the three month timeframe for the inception phase was appropriate. A number of other factors affected the development and implementation of CSO projects (the delays to Fund implementation and the CSO projects selected). Ultimately the inception phase length will be a question for DFAT WSH and what it would like to achieve with the next design. Inception phase length would depend on how high that Fund intends to continue to ‘raise the bar’ in WASH practice and the planned length of the Fund’s next phase (if the Fund’s next phase is to be shorter in duration, it would not make sense to have a longer inception phase). If more of a ‘partnership’ approach is taken to the development of Fund-wide indicators, then a longer inception phase may also be suitable to allow this process.

4.2.3 Suggested removal of ‘stop-go’ option The end of the inception phase was originally intended to be a stop-go point; that is, CSOs whose Operational Plans did not meet a certain standard at the end of the inception phase would not receive funding for their projects. CSOs reported that this created uncertainty. It was difficult for CSOs to make and keep commitments to partner government and to other local actors if CSOs did not know whether their project would be funded after the inception phase. DFAT WSH also had no appetite to utilise the ‘stop-go’

20 Note that six CSOs completed the inception phase and commenced implementation in August 2013, as original scheduled when the Fund was set up. The other CSOs did not complete the inception phase prior to the caretaker period and Australian Government election in August-September 2013. These CSOs did not commence implementation until May 2014. mechanism unless extreme circumstances warranted it. The evaluation team suggests that the inception phase no longer operate as a ‘stop-go’ point. With the removal of the ‘stop-go’ option, CSOs could be engaged and managed under a single grant agreement, rather than entering into separate grant agreements for the inception and implementation phases. This could create additional efficiencies for Fund management.

4.3 Conclusion A lot of effort went into conceiving and undertaking CSO selection and many of its elements were strong. In hindsight however, various aspects could be improved to help ensure the most suitable CSOs and projects are selected. These include: Incorporating the Fund’s theory of change in the design Providing greater weighting to key criteria and to a CSO’s past experience Possibly taking into account scores of both the CSO’s organisational capacity and project proposal to determine the final assessment Drawing upon a selection panel with appropriate yet diverse expertise – including in gender and disability inclusive development, and Ensuring appropriate dispute resolution processes are in place to resolve any significant differences between panel members’ scores. Fund stakeholders appreciated the Fund’s inception phase and the evaluation team suggests it be retained for the next Fund. An improvement would be to remove the ‘stop-go’ option. v.Glossary

ADRA Australian Research Development Awards ANCP Australian NGO Cooperation Program AACES Australia-Africa Community Engagement Scheme ANGO Australian Non-Government Organisation CSO Civil Society Organisation DFAT Department of Foreign Affairs and Trade FMF Fund Management Facility FTE Full-Time Equivalent I&I Innovation and Impact Fund IEC Independent Evaluation Committee K&L Knowledge and Learning KALM Knowledge and Learning Manager KPIs Key Performance Indicators KPQs key Performance Questions M&E Monitoring and Evaluation MERP Monitoring, Evaluation and Review Panel NGO Non-Government Organisation ODE Office of Development Effectiveness PRT Project Reporting Tool WASH Water, Sanitation and Hygiene WSH Water, Sanitation and Hygiene Section (DFAT) vi.References

Australian Agency for International Development, ‘Minute: Selection Process for the Civil Society WASH Fund’, unpublished, October 2012. Australian Agency for International Development, ‘AusAID Civil Society Water, Sanitation and Hygiene Fund – Guidelines and Templates’, July 2012. Australian Agency for International Development, ‘Civil Society Water, Sanitation and Hygiene Fund: Independent Completion Review’. June 2012. Australian Agency for International Development, ‘Water, Sanitation and Hygiene Global Programs and Cross-Regional Support: Civil Society WASH Fund Design Document’, May 2012. Australian Agency for International Development, ‘Civil Society Water, Sanitation and Hygiene Fund: Independent Progress Report’, February 2011. Civil Society Water, Sanitation and Hygiene Fund Management Facility, Annual Plans: CS WASH Fund, unpublished (2013-14 – 2015-16). Civil Society Water, Sanitation and Hygiene Fund Management Facility, Quarterly Reports, unpublished (Nos. 1- 12). Civil Society WASH Fund, ‘M&E Note 1: About the Civil Society WASH Fund’, July 2016. Civil Society WASH Fund, ‘M&E Note 2: Actor Analysis’, July 2016. Civil Society WASH Fund, ‘M&E Note 3: Approach Mapping’, July 2016. Civil Society WASH Fund, ‘M&E Note 4: Analysis of Implementation Progress’, July 2016. Civil Society WASH Fund, ‘M&E Note 5: Expected Changes and Emerging Impacts’, July 2016. Civil Society WASH Fund, ‘M&E Note 6: Analysis of Changes’, July 2016. Civil Society WASH Fund, ‘M&E Note 7: Summary of Monitoring Visits’, July 2016. Civil Society WASH Fund, ‘M&E Note 8: Behaviour Change’, July 2016. Civil Society WASH Fund, ‘M&E Note 9: Gender and Social Inclusion’, July 2016. Civil Society Water, Sanitation and Hygiene Fund Management Facility, Civil Society Water, Sanitation and Hygiene Fund website. Available from: . Commonwealth of Australia, Agreement Number 64839: Contract between Australian Agency for International Development and AID-IT Solutions Pty Ltd for AusAID Civil Society WASH Fund Monitoring, Evaluation and Review Panel, unpublished, October 2013 and Deeds of Amendment (Nos. 1 – 2). Commonwealth of Australia, Services Order No. 67478/1 for the Civil Society WASH Fund Monitoring, Evaluation and Review Panel Team Leader (Global) between AusAID and Griffin NRM Pty Ltd, unpublished, September 2013 and Deed of Amendment (No. 1). Commonwealth of Australia, Services Order No. 67821/1 for the Civil Society WASH Fund Monitoring, Evaluation and Review Panel between AusAID and FH Designs Pty Ltd, unpublished, September 2013 and Deed of Amendment (No. 1). Commonwealth of Australia, Agreement Number 65627: Contract between Australian Agency for International Development and GRM International Pty Ltd for Civil Society Water, Sanitation and Hygiene Fund Management Facility, unpublished, March 2013 and Deeds of Amendment (Nos. 1 – 5). Department of Finance, ‘Commonwealth Grants Rules and Guidelines’, July 2014. Department of Foreign Affairs and Trade, ‘The Australia Africa Community Engagement Scheme: Effective Partnerships for Sustainable Development Program Review 2011 – 2016’, May 2016. Department of Foreign Affairs and Trade, Quality at Implementation Report: 2013 Report for Civil Society WASH Fund, unpublished, March 2014. Department of Foreign Affairs and Trade, Aid Quality Check for INK182, Civil Society, Water, Sanitation and Hygiene Fund, unpublished, April 2015. Monitoring, Evaluation and Review Panel, ‘Performance Assessment Arrangements: CS WASH Fund II’, May 2013. Office of Development Effectiveness (Department of Foreign Affairs and Trade), ‘Evaluation of the Australian NGO Cooperation Program’, August 2015. vii. Annex A: Summary of CSO projects

Source: CS WASH Fund Annual Plan 2015-16 CS WASH Fund – CSO grantees Agreement dates Project Location/s Implementation Grant value21

Australian Red Cross (ARC) 1 May 2014 – Bangladesh $7,585,117 30 June 2018 Nepal Lesotho Concern Universal (Concern) 1 May 2014- Malawi $4,579,663 30 June 2018 Habitat for Humanity (HfH) 1 May 2014 – Bangladesh $1,070,072 30 June 2018 International Development 1 May 2014 – Cambodia $8,132,075 Enterprises (iDE) 30 June 2018 Vietnam International Rescue Committee 12 August 2013 – Pakistan $3,194,142 (IRC) 31 October 2017 Live & Learn Environment 1 May 2014 – Fiji $9,523,400 Education (LLEE) 30 June 2018 Papua New Guinea Solomon Islands Vanuatu Plan International Australia (Plan) 12 August 2013 – Indonesia $11,884,356 31 October 2017 Vietnam Pakistan Malawi Save the Children Australia (SCA) 1 May 2014 – Myanmar $2,308,043 30 June 2018 SNV Netherlands Development 12 August 2013 – Bhutan $4,103,131 Organisation (SNV) 31 October 2017 Nepal Thrive Networks Global (Thrive – 12 August 2013 – Cambodia $ 10,201,408 formerly East Meets West Foundation) 31 October 2017 Vietnam 1 May 2014 – 30 Lao PDR June 2018 (for Lao PDR) WaterAid Australia (WaterAid) 12 August 2013 – Papua New Guinea $11,282,827 31 October 2017

21 Note that CSOs received some additional funds for the inception phase. CS WASH Fund – CSO grantees Agreement dates Project Location/s Implementation Grant value

Timor-Leste Mozambique Welthungerhilfe (WHH) 12 August 2013 – Zimbabwe $9,050,000 31 October 2017 World Vision Australia (WV) 1 May 2014 – Sri Lanka $10,228,934 30 June 2018 Papua New Guinea Zimbabwe

Impact and Innovation Fund CSO grantees Project Location/s I&I Fund grant value

iDE Cambodia $200,000 WaterAid PNG and Timor-Leste $194,425 Welthungerhilfe Zimbabwe $199,995 Plan Australia Vietnam $99,570 SNV Netherlands Bhutan $100,000 Development Organisation Annex B: Fund management: Expected roles and information flows viii. Annex C: Evaluation methods

The evaluation addressed the following evaluation questions.

Priority questions (approximately 80% of the evaluation) M To what extent are the Fund’s management arrangements appropriate, effective and efficient? M(i) To what extent are DFAT staff in Canberra and at posts, Fund Management Facility staff, MERP members, and CSO staff working effectively together to achieve Fund objectives? M(ii) To what extent are contracting arrangements facilitating the effective and efficient attainment of Fund objectives? M(iii To what extent have the M&E and learning approaches inherent in the Fund’s design and implementation assisted to ) develop and achieve better outcomes? M(iv) To what extent is Fund management cost-effective? M(v) To what extent do the Fund’s management arrangements give due regard to advancing equity and inclusiveness, particularly in relation to gender and disability? Other questions (approximately 20% of the evaluation) S To what extent did the two-part CSO selection process enable selection of the most effective CSOs to deliver WASH projects? S(i) To what extent was CSO selection (including the inception phase) consistent with the design’s guidance and intent? S(ii) To what extent did CSO selection give due regard to advancing equity and inclusiveness, particularly in relation to gender and disability?

Methods The methods used to address each evaluation question are outlined below and summarised at Appendix B. The evaluation methods included key informant interviews, an online survey and document review. Financial and staff number data were also be used. Note that non-participant observation of a regional learning event was planned in the draft evaluation plan, however, this was not undertaken as the evaluation team was unable to obtain the necessary visa. All evidence was triangulated where feasible to ensure a degree of rigour. So far as possible major evaluation findings drew upon the analysis of multiple data sources and were tested with Fund stakeholders. The evaluation used a ‘recommendations workshop’ to develop evaluation recommendations. Following development of the draft evaluation report, an externally facilitated workshop with the WSH team, the MERP, Fund Management Facility staff and the evaluation team was held. The purpose of the workshop was to develop and agree on the recommendations that flow from the evaluation findings. This process aimed to ensure the evaluation recommendations were practical, can feasibly be implemented, and have support from the Fund’s management team.

Evaluation question: To what extent are the Fund’s management arrangements appropriate, effective and efficient? The evaluation focussed on understanding the Fund’s management arrangements and the extent to which they have enabled CSOs to develop and implement high quality projects. To do this, the evaluation defined the behaviours and communication flows we would expect to see if the management arrangements support CSOs to deliver high quality projects. Data was then gathered to assess the extent to which these behaviours and communication flows were taking place. The approach taken to address each sub-question is further explained below. Note that these questions focused on the current management arrangements and how they are functioning. Of key interest to the evaluation was how the management arrangements could or should be adapted in light of the changed context. This was a key element of data analysis, with the evaluation report outlining how well the management arrangements are currently working and how they may be adapted to better suit the changing context.

Question M(i): To what extent are DFAT staff in Canberra and at posts, Fund Management Facility staff, MERP members and CSO staff working effectively together to achieve Fund objectives? To address this question, the behaviours and communication channels between Fund stakeholders which we would expect to see if Fund stakeholders are working effectively together were defined. Drawing on Fund documents, as well as initial discussions with Fund stakeholders, an outline of the behaviours and communication lines is at Annex B to this report. Note that this diagram is only a broad outline; many day to day practices have not been included, while some communication channels which involve multiple parties or steps have been simplified. Annex B highlights a number of features we would expect to see if the management arrangements are functioning well. These include: A focus on ensuring CSOs are supported through ongoing learning and M&E to deliver high quality projects Strong communication between DFAT WSH, the Fund Manager and the MERP to ensure good Fund management decisions are made. The evaluation particularly focused on whether the management arrangements and information provided by the MERP and FMF meets DFAT’s needs and enables DFAT to make informed decisions on Fund management Good links between the Fund Manager, DFAT Posts and CSOs to ensure DFAT-supported WASH projects in each country are coordinated, and that DFAT Posts can draw on relevant Fund information to inform discussions with host governments Good links between the Fund Manager, the MERP and DFAT Posts to ensure M&E data is available to, and aligns with, host government M&E systems. The evaluation assessed the extent to which these features were being implemented to identify Fund management strengths as well as areas that can be improved. This was achieved through: Review of relevant program documents, particularly Aid Quality Checks (AQCs), contracts and regular reporting from the FMF Interviews with DFAT (WSH and Posts), the MERP, the FMF, and CSO program managers, and An online survey of CSO project managers.

Question M(ii): To what extent are contracting arrangements facilitating the effective and efficient attainment of Fund objectives? The DFAT WSH team holds the main contracts for the Fund. This includes funding agreements with each of the 13 CSOs, one contract with Palladium for the Fund Management Facility, and a contract with each of the three MERP members – a total of 17 contracts. An initial review of the Fund documents indicates WSH chose to manage the procurement processes to ensure quality outcomes were achieved22. The documents reviewed provide little further justification for the contracting model, including why the MERP and Fund Management Facility were contracted separately. The contracting arrangement creates a significant amount of administrative work for WSH, who must ensure contracts are maintained, payments are made and partner performance assessments are completed. The contracting arrangements also mean there is potential for confused communication and authority. For example, although WSH has contracts with each of the CSOs, the Fund

22 See minute ‘Change in method for MERP’, 12 August 2012. Management Facility manages the Fund’s day to day operations and is the first point of contact for CSOs. The Fund Management Facility and the MERP are also expected to work closely together without having a contractual arrangement in place. Key informant interviews with WSH, the Fund Manager, the MERP and CSO program managers were used to explore the extent to which these contracting arrangements support achievement of the Fund’s objectives. Interviews focused on the advantages and disadvantages of the current arrangements and how they could be improved. Interview data was supported by document review, particularly of the contracts in place.

Question M(iii): To what extent have the M&E and learning approaches inherent in the Fund’s design and implementation assisted to develop and achieve better outcomes? To address this question the evaluation team focused on the extent to which the MERP and the Knowledge and Learning component have assisted CSOs to improve their projects. It looked at the relationships and communication flows between CSOs, the MERP and the KALM, as shown in Annex B to this report, in some depth. Data collection particularly focused on: The extent to which MERP monitoring visits and involvement in project design, M&E and reporting has assisted CSOs to improve their projects The extent to the approaches to sharing knowledge and learning, such as electronic media and learning events, have assisted CSOs to improve their projects The extent to which the MERP and the KALM have collaborated to support CSOs. The evaluation team also examine the volume of evaluation data collected and the extent to which is it useful for all Fund stakeholders, and used by them to inform program-wide or project-specific management decisions. Data was collected through an electronic survey of CSO project managers (see Appendix D). It was also discussed in key informant interviews, particularly with CSO program managers, the MERP and the KALM. The volume and use of M&E data was covered in interviews with DFAT staff and will be assessed through examination of the Fund’s online database.

Question M(iv): To what extent is Fund management cost-effective? This question considers whether the Fund’s program management costs are proportionate to the effectiveness of the Fund’s program management arrangements. Note that as this evaluation is not assessing the effectiveness of the Fund projects, the evaluation did not examine the broader issue of whether the Fund has delivered value for money. To explore this question, the evaluation benchmarked the Fund’s management costs and management effectiveness against two other similar CSO programs: the Australia NGO Cooperation Program (ANCP) and the Australia Africa Community Engagement Scheme (AACES). These programs were chosen because they have similar funding mechanisms to the CSO WASH Fund. In addition, they have both recently been independently evaluated, meaning significant additional data collection on them was not required. The evaluation approach was as follows: The program management costs for the three programs (the Fund, ANCP and AACES) was collected. This included the number of DFAT and contractor staff involved, as well as expenditure on program management. The Fund’s cost were then compared to the other programs’ costs to determine whether its costs are low, moderate or high The effectiveness of the program management arrangements (that is, the extent to which management arrangements supported CSOs to deliver high quality projects) for each program were examined. For the Fund, this wsd achieved through examination of the evaluation questions outlined above. For ANCP and AACES, it wsd achieved by examining their existing evaluations. The effectiveness of the Fund’s program management arrangements was then be compared to the other programs’ to determine whether management effectiveness is low, moderate or high The resulting ratings for the Fund’s program management costs and effectiveness were assessed against the following matrix. This allowed the evaluation team to draw conclusions on whether the Fund’s management costs and effectiveness are proportionate and cost-effective or not.

Table 1: Cost effectiveness matrix Fund management costs are: Fund management effectiveness is: Is the Fund cost-effective?

High Cost-effective

High Moderate Cost-effectiveness could be improved

Low Not cost-effective

High Very cost-effective

Moderate Moderate Cost-effective

Low Cost-effectiveness could be improved

High Very cost-effective

Low Moderate Very cost-effective

Low Not cost-effective23

Question M(v): To what extent do the Fund’s management arrangements give due regard to advancing equity and inclusiveness, particularly in relation to gender and disability? Given the high priority placed on gender equality and disability inclusiveness by the aid program, the evaluation examined the extent to which the Fund’s management arrangements have ensured these topics have been integrated into the Fund’s work. Data was collected in the following ways: Content analysis was conducted to assess the extent to which gender equality and disability inclusion are integrated into key documents. Analysis particularly focused on: the Fund’s Performance Assessment Framework; regular reporting from the Fund Management Facility and CSOs; Aid Quality Checks completed by DFAT; and documents and discussion on the Fund’s website The survey of CSOs included questions on the extent to which the integration of gender equality and disability inclusion has been encouraged by the Fund.

Evaluation question: To what extent did the two-part CSO selection process enable selection of the most effective CSOs to deliver WASH projects?

Question S(i): To what extent was CSO selection (including the inception phase) consistent with the design’s guidance and intent? To address this question, the team compared the design’s intent with actual implementation. The Fund’s design had particular views of CSO strengths in the WASH sector and areas of comparative CSO weakness. As per the Fund’s design, three broad areas are selected for initial comparison: (1) the types of CSOs and proposals intended to be selected and actually selected; (2) the intent and actual implementation of the Fund’s inception phase; and (3) the intended and actual geographical balance of projects supported by the Fund. Data was collected in the following ways:

23 If the Fund has low fund management costs and low fund management effectiveness, it could be considered cost-effective because costs and effectiveness are proportionate. However, low fund management effectiveness is not a desirable outcome and it is the evaluation team’s view that a program can’t be cost-effective if it is not effective to begin with. A document review focussed on the design document, DFAT minutes concerning CSO selection, and Fund monitoring reports during inception A review of a small number of original CSO proposals to examine how they were assessed and scored by selection panel members Interviews with a range of respondents involved in the Fund’s design (WSH section and MERP members), and with those involved in the CSO selection process and the inception phase (DFAT selection panel members, WSH, MERP, FMF and funded CSOs’ program managers). An online survey of CSOs which applied but which were unsuccessful in obtaining Fund support to better understand whether the selection phase was perceived as transparent.

Question S(ii): To what extent did CSO selection give due regard to advancing equity and inclusiveness, particularly in relation to gender and disability? This question investigated the degree to which there is variation in the standard of how funded CSOs treated gender and disability, both in their initial proposals and then during the inception phase, and how this was assessed by the Fund. A uniformly high standard of attention to gender and disability inclusion within the initial CSO proposals by funded CSOs would indicate that funded CSOs are already strong in these areas (provided the selection process itself was robust). A uniformly high standard of attention only following the inception phase could indicate a successful feature of that phase. The team also investigated the extent to which the CSO selection panel and Fund Management stakeholders supported CSOs during the inception phase to appropriately consider gender and disability inclusion. Data collection particularly focused on relevant documents (CSO proposals, DFAT records regarding the selection process, Fund monitoring reports, and best practice guidance for gender and disability inclusion) and interviews.

Ethical issues Consistent with DFAT’s draft Evaluation Policy, the evaluation will be conducted consistent with the AES Guidelines for the Ethical Conduct of Evaluations. The following three ethical issues are noteworthy. Informed consent will be sought from all participants who provide data for the evaluation through interview or survey response. Data will be securely maintained using DFAT’s systems. Interview and survey data will only be accessible to the evaluation team and other parts of DFAT will not have direct access to it. This will ensure respondents can comment frankly and freely. Interviewees can also request transcripts of their interview if desired. In the evaluation report all data will be de-identified (that is, responses will not be attributed to the individual who gave them). However, a number of groups will have a very small number of respondents; for example, there are only three MERP members and only two FMF staff will be interviewed. This means that it may be possible for those reading the evaluation report to link a particular response to a particular individual. All respondents will be made aware of this issue and will be given the option not to proceed with interview. It is also possible that the evaluation findings will have adverse findings for groups such as the MERP, FMF or the DFAT WSH section. For example, the evaluation could suggest that fewer contractor or DFAT resources are required to manage the future phase. As contractors and DFAT WSH will be involved in developing recommendations for the evaluation, ethical tensions could arise for these groups. In such a case: All groups will be asked to set aside personal interests and act in the interests of the program The evaluation team will reserve the right to make the final decision on evaluation recommendations. Appendix A: Evaluation matrix

No. Evaluation Question Primary Audience Data collection Data analysis Tools M To what extent are the Fund’s management arrangements appropriate, effective and efficient?

M To what extent are DFAT staff in Canberra and at DFAT (WSH and Post), Key informant interviews Data mapped against the expected Interview (i) posts, Fund Management Facility staff, MERP FMF, MERP and CSOs – with DFAT (WSH and behaviours outlined in Annex C to guides members, and CSO staff working effectively to inform the remainder of Post), MERP, FMF and highlight strengths and areas for together to achieve Fund objectives? the Fund CSO program managers improvement Survey

DFAT WSH – to inform Online survey of CSO design of the next Fund project managers phase Document review, particularly Aid Quality Checks, contracts and regular reporting from FMF

M To what extent are contracting arrangements DFAT WSH – to inform Key informant interviews Qualitative analysis to identify the Interview (ii) facilitating the effective and efficient attainment design of the next Fund with WSH, FMF, MERP advantages and disadvantages of the guides of Fund objectives?24 phase and CSO program current arrangements and possible managers improvements for the next phase

Document review, particularly contracts M To what extent have the M&E and learning DFAT WSH, MERP and Online survey of CSO Data mapped against the expected Survey (iii) approaches inherent in the Fund’s design and KALM – to inform the project managers behaviours outlined in Annex C to implementation assisted to develop and achieve remainder of the Fund highlight strengths and areas for Interview

24 Note: QAIs/AQCs don’t mention any concerns for DFAT on the contract management load. Note that DFAT has to do partner performance assessments for all the CSOs plus Palladium better outcomes? Key informant interviews improvement guide DFAT WSH – to inform with MERP, KALM, CSO design of the next phase program managers and DFAT

Examination of Fund’s online database (known as Bob)

M To what extent is Fund management cost- DFAT WSH – to inform FTE and contract costs for Comparison of Fund, ANCP and Cost- (iv) effective? design of the next phase the Fund, ANCP and AACES management costs to determine effectiveness AACES if Fund costs are low, moderate or high matrix

ODE’s ANCP evaluation Comparison of Fund, ANCP and (2015) AACES management effectiveness to determine if Fund management Final evaluation of effectiveness is low, moderate or high AACES program (2016) Mapping Fund ratings against the cost- effectiveness matrix M To what extent do the Fund’s management DFAT WSH, FMF, MERP Document review, Qualitative analysis to the strengths and Survey (v) arrangements give due regard to advancing and CSOs – to inform the particularly Fund PAF, weaknesses of the current approaches equity and inclusiveness, particularly in relation remainder of the Fund regular reporting from the and possible improvements for the next to gender and disability? FMF and CSOs, AQCs phase DFAT WSH – to inform and website documents design of the next phase Online survey of CSO project managers

S To what extent did the two-part CSO selection process enable selection of the most effective NGOs to deliver WASH project?

S (i) To what extent was CSO selection, (including the DFAT WSH – to inform Document review, Qualitative comparison to analyse Interview inception phase) consistent with the design’s design of the next phase particularly of design extent to which intended outcomes guides guidance and intent? documents, to establish the regarding CSO selection (noted in intent of the design Design guidance and intent) and CSO Survey of selection process as implemented are CSOs consistent Key informant interviews with selection panel members (where available), WSH, MERP, Fund Management Facility, and program managers of funded CSOs

Online survey of unsuccessful CSOs

S To what extent did CSO selection give due DFAT WSH – to inform Document review Qualitative analysis to identify the Interview (ii) regard to advancing equity and inclusiveness, design of the next phase strengths and weaknesses of the current guides particularly in relation to gender and disability? Key informant interviews approaches and possible improvements with selection panel for the next phase members (where available), WSH, MERP, Fund Management Facility, and program managers of funded CSOs. Appendix B: CSO survey The Office of Development Effectiveness (ODE) is evaluating the management arrangements for the Civil Society Water, Sanitation and Hygiene Fund (the Fund). The focus of the evaluation is the main managers of the Fund – the Department of Foreign Affairs and Trade (DFAT), the Fund Management Facility (FMF) and the Monitoring, Evaluation and Review Panel (MERP) – and how they interact with each other and with CSOs. The evaluation aims to identify whether the management arrangements are efficient and appropriate, and how they can be improved in the future. This survey is part of the evaluation. It will help us better understand how the Fund’s management arrangements have supported civil society organisations (CSOs) to implement high quality WASH projects. Survey responses will be reported in aggregate and will not be attributed to you or your organisation. The survey will take around 15 minutes to complete. For staff of CSOs which applied to the Fund but were not successful, the survey will take around five minutes to complete. On behalf of ODE, thank you for your participation.

A. Background information 1. Where is your CSOs’ headquarters? o Australia o outside Australia 2. Approximately how many staff are employed by your CSO? (include all staff working in all countries and headquarters staff) o More than 100 o less than 100 3. When applying for funding from the CS WASH Fund, was your CSO: o Successful? (if so, please answer all sections of the survey) o Unsuccessful? (if so, please answer only Section B, Selection Phase)

B. Initial selection of CSOs DFAT (formerly AusAID) called for CSOs to provide submissions for the Fund in June 2012. 4. Were you involved in developing your CSO’s submission to the Fund? o Yes o No (please go to Section C, Inception phase) 5. In relation to the call for submissions, to what extent would you agree to the following statements: Strongly Disagree Neither agree Agree Strongl disagree or disagree y agree I had sufficient information on the Fund’s requirements to develop a high quality submission I understood how submissions would be assessed and how decisions on funding would be made

6. If your CSOs’ submission was not successful, were you given feedback on why the submission did not receive funding? o Yes o No o Unsure if feedback was received o N/A (submission was successful) 7. If you answered Yes, was it clear from the feedback why your CSO’s submission did not receive funding? [free text; leave blank if can’t recall] o Yes o No o Unsure/can’t recall 8. Do you have any comments on the selection phase which you would like to share? [free text]

C. Inception Phase These questions relate to the Fund’s inception phase, when selected CSOs were provided with funding and support to develop operational plans (or designs) for their projects. 9. How were you involved in the inception phase and the development of your project’s design/operational plan? (check all that apply) o Not involved (please go to section D, Engagement with the MERP) o Participated in the Inception Phase Design Workshop in Canberra in April 2013 o Received advice from the Fund Management Facility and/or the Monitoring, Evaluation and Review Panel (MERP) o Other (please specify) 10. Overall, to what extent did the support provided by the Fund help you improve your design/operational plan: 1 2 3: 4 5 N/A Not at all Slightly Moderately Significantl Extremely y Inception Phase Design Workshop Advice from Fund Management Facility and/or MERP Advice on gender equality issues Advice on disability inclusiveness issues Other (please specify)

D. Fund M&E and engagement with the Monitoring, Review and Evaluation Panel (MERP) These questions relate to the Fund’s monitoring and evaluation system your engagement with the Monitoring, Review and Evaluation Panel: Bruce Bailey, Paul Crawford and Paul Tyndale-Biscoe. 11. To what extent do you agree with the following statements: Strongly Disagree Neither agree or Agree Strongl disagree disagree y agree The M&E reports (e.g. 6 monthly reports) we provide to the Fund Management Facility and DFAT are also useful to our organisation. The M&E reports (e.g. 6 monthly reports) we provide to the Fund Management Facility and DFAT help us make improvements to our project The level of reporting required for our project is overly onerous The Fund’s monitoring and evaluation system adds more cost to the way we do things without improving our project The Fund’s monitoring and evaluation system helps our project identify learnings which can be shared across the Fund The Fund’s M&E arrangements enable my project to adapt and change to accommodate learnings during implementation The requirement for a Gender equality and social inclusion (GeSI) operational plan and related Key Performance Indicators and Questions has strengthened my project’s approach to gender equality The requirement for a Gender equality and social inclusion (GeSI) operational plan and related Key Performance Indicators and Questions has strengthened my project’s approach to disability inclusiveness The actor-centred theory of change (delivery teams, change agents and target populations) has helped my project to identify and influence key actors important to WASH service delivery in my location(s). Monitoring trips conducted by MERP members were professional and constructive and provided opportunities to reflect and improve the design and implementation of my project. The MERP model of having independent specialists retained for the whole life of the Fund is useful and helps to improve project implementation, monitoring, review and reporting.

12. How have you worked with the MERP? (check all that apply) o I have not received any support from the MERP [go to next section] o I was involved in a monitoring visit conducted by a MERP member o I have received support to develop my project’s operational plan o I have received support for project reporting o I have received other M&E support, including capacity building o Other: please specify 13. Overall, to what extent has support from the MERP in the following areas helped you improve your WASH project? 1 2 3: 4 5 N/A Not at all Slightly Moderately Significantl Extremely y Monitoring visit Support for operational plan Support for project reporting Other M&E support, including capacity building Other – please specify

In relation to the monitoring visit by a MERP member: 14. What aspects of the monitoring visit were most helpful for improving your project? [free text] 15. What aspects of the monitoring visit were least helpful for improving your project? [free text] 16. Would further monitoring visits be useful? Yes/No/Unsure Please explain your answer [free text]

E. Engagement with the Knowledge and Learning Component The Fund-wide Knowledge and Learning Component is implemented by a team at the Fund Management Facility led by the Knowledge and Learning Manager, Bronwyn Powell. 17. What knowledge and learning opportunities facilitated by the Fund have you participated in? o Watched webinar hosted on CS WASH Fund website o Contributed to or followed e-discussion/forum hosted on cswashfund.org o Accessed resources or information hosted on cswashfund.org o Attended a regional or global learning event o Drawn on a research project (for example, an Australian Development Research Award (ADRAs)) o Collaborated with another CSO supported by the Fund o Developed and/implemented your organisation’s Knowledge and Learning Plan o Received feedback on my organisation’s Knowledge and Learning Plan from the Knowledge and Learning Manager o Received newsletter/electronic updates o Watched videos shared by CS WASH Fund o Participated or followed Civil Society Innovation Award o Accessed resources and information shared via CS WASH Fund social media channels (Facebook, Google +, LinkedIn Group, Twitter) o Member of Knowledge and Learning Advisory Group o Completed survey or interview for CSO Learning for Impact research with Institute for Sustainable Futures UTS o Other (please specify): [free text] 18. In general, when engaging in Fund-wide learning and knowledge activities how frequently are the following issues discussed? Service 1 2 3: 4 5 N/ A Never Rarely Occasionally Frequently Very frequently Gender equality Disability inclusiveness

19. To what extent has engagement with these knowledge and learning activities helped your project delivery team improve its knowledge of issues related to WASH? 1 2 3: 4 5 N/ A Not at all Slightly Moderately Significantl Extremely y Webinar hosted on CS WASH Fund website E-discussion/forum hosted on cswashfund.org Accessed resources or information on cswashfund.org Regional or global learning event Linked to a research project Collaborated with another CSO Developed and/or implemented Knowledge and Learning Plan Received feedback on your organisation’s Knowledge and Learning Plan from the Knowledge and Learning Manager Received newsletter/electronic updates Watched videos shared by CS WASH Fund Participated or followed Civil Society Innovation Award Accessed resources and information shared via CS WASH Fund social media channels (Facebook, Google +, LinkedIn Group, Twitter) Member of Knowledge and Learning Advisory Group Other (please specify)

20. To what extent has engagement with these knowledge and learning activities helped your project delivery team improve practices in your WASH project? 1 2 3: 4 5 N/ A Not at all Slightly Moderately Significantl Extremely y Webinar hosted on CS WASH Fund website E-discussion/forum hosted on cswashfund.org Accessed resources or information on cswashfund.org Regional or global learning event Linked to a research project Collaborated with another CSO Developed and/or implemented Knowledge and Learning Plan Received feedback on your organisation’s Knowledge and Learning Plan from the Knowledge and Learning Manager Received newsletter/electronic updates Watched videos shared by CS WASH Fund Participated or followed Civil Society Innovation Award Accessed resources and information shared via CS WASH Fund social media channels (Facebook, Google +, LinkedIn Group, Twitter) Member of Knowledge and Learning Advisory Group Other (please specify)

21. To what extent do you agree with the following statements: Strongly Disagree Neither agree or Agree Strongl disagree disagree y agree Fund-wide learning and knowledge activities are useful to our organisation Fund-wide learning and knowledge activities help us make improvements to our project The time and resources required to engage with Fund-wide learning and knowledge activities is overly onerous Fund-wide learning and knowledge activities add more cost to the way we do things without improving our project The Fund’s approach to learning and knowledge helps our project identify learnings which can be shared across the Fund and with the global WASH sector Fund-wide channels such as the website, social media and the newsletter have helped our CSO disseminate and promote our knowledge and learning outputs The Fund’s model of having a Fund-wide knowledge and learning component led by the Knowledge and Learning Manager is useful and helps us improve our project Feedback from the Knowledge and Learning Manager on our regular reporting has been valuable to improve our project and knowledge and learning activities Fund-wide learning and knowledge activities have helped us improve our project’s approach to gender equality Fund-wide learning and knowledge activities have helped us improve our project’s approach to disability inclusiveness Our project delivery team collaborated or learnt from other CSOs because of the Fund’s learning and knowledge activities The Fund’s approach to learning and knowledge has helped our project delivery team link to and draw on useful research The ideas, themes and topics emphasised through Fund-wide knowledge and learning activities have been relevant and current for our project delivery team The ideas and content on WASH provided through Fund-wide knowledge and learning activities has extended our delivery team to consider good practice and new approaches The ideas and content on knowledge and learning management provided through Fund-wide knowledge and learning activities has extended our delivery team to consider good practice and new approaches Our project delivery team has had the opportunity to help shape Fund-wide knowledge and learning activities, particularly global and regional learning events The Fund’s knowledge and learning component has helped our organisation in activities beyond the Fund.

F. Engagement with DFAT Posts and Host Governments These questions relate to your engagement with DFAT Posts (embassies or high commissions) in the country where your project is implemented. 22. How frequently do you engage with DFAT Post staff in the country where your project is implemented? Chose the answer which is most correct Never Rarely Occasionally Frequently Very (1-2 (monthly) frequently times per (5-6 times year) per year) (weekly)

23. Is this level of engagement: Not About Too much? enough? right?

24. In your project, do the “change agents” you work with include government officials at either national or sub-national levels? o Yes (please go to question 28) o No 26. If no, how frequently do you engage with the host government (at either national or sub-national level) in the country where your project is implemented? Never Rarely Occasionally Frequently Very (1-2 (monthly) frequently times per (5-6 times year) per year) (weekly)

27. Is this level of engagement: Not About Too much? enough? right?

28. Is M&E data from your project (eg number of activities implemented; number of beneficiaries reached) shared with the host government at either national or sub-national level in the country where your project is implemented? Yes/No/Unsure 29. If yes, how does this take place? [free text] 30. If no, why is this data not shared? [free text Appendix C: Interview guides The interview guides below are designed for use as semi-structured guidance tools for interviews with respondents. These draft guides will be tailored further for interviews with specific respondents to understand and synthesise perceptions about roles, interactions and Fund processes. Not all questions will be asked, or will be asked verbatim. Final questions posed to specific respondents will also build on analysis derived from the document review and other respondent interviews. Questions will cover the most salient topics in the timeframe allowed. After the completion of the document review, a further interview guide will be developed to inform the interviews of authors of recent evaluations of other DFAT-funded CSO programs (for example, the ANCP and AACES programs).

For Respondents responsible for Fund Management: DFAT WSH Section, MERP, and the Fund Management Facility’s Fund Manager and KALM Code: Question Key Evaluation Question M(i) How would you describe your role in the Fund?

M(i) Describe your interactions with the following: [Note: not all the roles listed below will apply. Questions will be tailored to the specific respondent.]:

- DFAT WSH Section - MERP - Fund Management Facility’s Fund Manager - KALM - CSOs (Program Managers and/ or Project Managers) - DFAT Posts - external stakeholders (eg. partner government and host country actors, WASH Reference Group, WASH research institutes). M Describe the strengths of the Fund’s management arrangements. Describe the weaknesses of the Fund’s management arrangements. M(ii) Describe the advantages and disadvantages of current contracting arrangements. For respondents involved with the Fund at the time of CSO selection and/ or the inception phase: M Describe your role with the Fund during the time of CSO selection and/ or inception. S(i) How would you describe the types of CSOs and projects that the Fund was intended to fund? S(i) Do you think the outcomes of the initial selection process matched the intent put forward in the design? S(i) What was the intended purpose of the inception phase? S(i), (ii) What were the benefits and draw-backs of the Fund’s inception phase, as implemented? S(ii) Describe to what extent the inception phase helped to improve CSO proposals’ treatment of gender and disability.

For respondents who became involved with the Fund following the inception phase:

S(i) How would you describe the types of CSOs and projects that the Fund supports? What do they have in common? S(ii) Describe whether there is great variation between CSOs’ treatment of gender and disability now. S(ii) Describe the strongest CSO performers with respect to gender and/ or disability inclusion and what their example could indicate for future CSO selection. M/S Anything we missed? Would you like to make any further comments?

For CSO Program Managers M(i) Describe your role and how long you have worked with the Fund. M(i) Describe your (and/or CSO Project Managers’) interactions with the following: [Questions will be tailored to the specific respondent.]:

- MERP - Fund Management Facility’s Fund Manager - KALM - DFAT WSH Section - other CSOs supported by the Fund - DFAT Posts - external stakeholders (eg. partner government and host country actors, WASH research institutes).

M Describe the strengths and weaknesses of the Fund’s management arrangements. How would you describe the quality of relationships between those involved in Fund Management [DFAT WSH/ MERP/ Fund Management Facility’s Fund Manager/ KALM] and funded CSOs? M(ii) Describe the advantages and disadvantages of current contracting arrangements. S(i), (ii) For respondents involved with the Fund at the time of the inception phase: S(ii) What were the benefits and draw-backs of the Fund’s inception phase, as implemented? Describe whether and how the inception phase helped your project to advance gender and/ or disability inclusion. M/S Anything we missed? Would you like to make any further comments?

For DFAT Posts that have ‘opted-in’ to the Fund M(i) Describe your role. M(i) Describe your experience with the Fund. M(i),(iii),(v) Tailored questions based on responses from other respondents and DFAT posts themselves. Describe your interactions with: - WSH Section - Fund Management Facility’s Fund Manager - funded CSOs. Describe whether you have used information generated from the Fund or funded CSO projects in your role, including related to gender and disability inclusion.

For Respondents on the Selection Panel that selected the Fund’s CSOs S(i) In what capacity were you selected to participate on the selection panel? Describe your role and experience on the selection panel. S(i) Describe the elements of the Fund’s CSO selection process that worked well and the elements that required strengthening. Describe to what extent you found the guidance for assessing CSO capacity and CSO projects useful and easy to use.

S(ii) To tailor questions following document review of CSO selection process: Describe how the panel assessed gender inclusiveness. S(ii) Describe how the panel assessed disability inclusiveness.