Understanding Monitoring and Evaluation - Fact Sheet

FOOD AND GARDEN ORGANICS BEST PRACTICE COLLECTION MANUAL: PLANNING YOUR SCHEME – FACTSHEET 8 – UNDERSTANDING MONITORING AND EVALUATION  Assess expenditure and Understanding control costs, in terms of anticipated quantity of organics collected, demand for liners, and the impact on education monitoring and and processing costs of different contamination levels evaluation  Evaluate return on investment to justify existing budgets or Monitoring and evaluation helps develop an persuade budget holders that more understanding of how a scheme or money is required to achieve statutory communications campaign is performing, and and local targets may identify opportunities for improvement.  Plan scheme expansions This applies to both waste management and design (or redesign) scheme so that services and the communications activities targets are met or exceeded undertaken to promote them.  Plan targeted ‘ Monitoring’ means regularly measuring communications to improve performance outcomes such as customer satisfaction,  Address the issues that are participation rates, contamination rates and really impacting on scheme success. diversion rates. ‘Evaluating’ means drawing conclusions from the monitoring data in terms When to monitor of how well the scheme is performing, or the If you want to monitor the effect of a service effect of the communication activity. Monitoring change, then you need to monitor both before and evaluation are therefore two distinct the service starts or the service is changed, and activities, with monitoring being impartial and again afterwards. The purpose of monitoring in factual while evaluation tends to be subjective advance is to establish a baseline from which and value laden. you can measure a change. Most monitoring can be done by someone who does not know the local area, while only Your aim someone who understands the context and An aim is a broad statement of what you are local environment can do the evaluation, for trying to achieve and there is usually one example someone who has knowledge of overarching aim. An example of a monitoring previous programs, socio-economics, aim would be ‘to measure the performance of demographics, and data. This means that, the organics service’: although you can commission somebody to measure what your service is doing, ultimately To capture xx% of the total generated organics you need to evaluate whether these outcomes in the kerbside organics bin by [xx date]. are good, satisfactory or poor. Your objectives

Monitoring and Objectives are a much more specific statement evaluation help you to: of what you are trying to achieve and it is common to have more than one objective for a  Measure customer monitoring and evaluation programme. You can satisfaction and user attitudes to demonstrate if you have achieved an objective. establish how these are impacting on the performance of your scheme There are three types of objectives that relate to  Measure progress against the measurement of specific inputs, objectives and targets, so you will know intermediate outcomes, or final impacts. in advance if you are likely to hit or miss them – for example less than 3% contamination, more than 80% participation, more than 60% diversion and 95% customer satisfaction  Identify successful systems as well as problems or performance issues, so that you can target your efforts to those neighbourhoods where improvements will make the most impact

1 Make your Objectives SMART The methods

There are various methods that can be employed to demonstrate the effectiveness of a scheme, including:

 Tonnage data analysis, for example increased tonnage of organics and decreased residual waste tonnage  Waste auditing per bin, or aggregated via a visual waste audit, or physical waste characterisation  Set out and participation rate monitoring, for example identifying 80% of all organics bins are put out for collection but only 50% of them contain food organics  Organics capture analysis  Stakeholder feedback Key Performance Indicators  Communication evaluation.

KPIs are quantifiable measures that capture critical success factors and are a framework for measuring achievements. They are presented as units of measurement (e.g. number, percentage, tonnage). They are the tools that enable you to monitor the success or your activities. KPIs allow you to convert your monitoring data into something usable and meaningful. Each objective that you set should have at least one related KPI.

Type of Example KPI objective Resource Recovery Officer tagging bin (Bankstown Input To distribute 10,000 Number City Council) leaflets by [xx date] to of leaflets 20,000 households on 1 Tonnage data analysis the kerbside organics The relevant processing facility can usually collection scheme provide information about daily yields of Impact To decrease the kg per organics. This data can be logged in a organics disposed in the household spreadsheet, with trends observed over time. garbage bin from xx per year This method can also be employed to monitor kg/hh/year to yy (kg/hh/ye the effectiveness of new communication kg/hh/year ar) campaigns as the scheme progresses, as well as identifying and targeting contamination by Example of objectives and KPIs measuring weight per truck run and comparing effectiveness between different areas.

2 FOOD AND GARDEN ORGANICS BEST PRACTICE COLLECTION MANUAL: PLANNING YOUR SCHEME – FACTSHEET 8 – UNDERSTANDING MONITORING AND EVALUATION 2 Waste auditing Participation rate provides similar information, but takes into account the fact that some A visual waste audit involves inspecting bins householders may not set out a collection and truck loads in order to estimate the container on a specific day, for example percentage of each waste type in the bin or because they are away on holiday or do not load. A limitation of a visual assessment is that have sufficient materials to put out for it does not allow for compaction of the waste, collection. It is defined as the number of which impacts on the accuracy of results. households within a target area that participate However, it is less time consuming (and in an organics collection at least once during therefore less expensive) than a physical waste the monitoring period (typically three audit. consecutive collections), divided by the total number of households within that area. The most detailed and robust waste data can be obtained by auditing. A household by No. households recorded as setting out at least once in a household audit, or aggregated sampling, will defined period enable you to measure household generation of No. households monitored in that period waste, recyclables and organics; determine the composition of each waste stream, and assist in Participation rate monitoring can be used to monitoring the performance of the scheme. identify non-participating households so they can be targeted for door to door It is recommended that an audit of the residual communications or promotional work to provide waste stream be conducted prior to the additional information on the scheme. Further implementation of the scheme. This will provide participation monitoring can be done after a you with baseline data for the total amount of campaign in order to understand the impact it waste produced, and the type and volume of has had on the scheme. organics currently in the residual waste stream. The residual waste stream should then be 4 Capture rate analysis audited, using the same metrics, seasonally (at least summer and winter) to obtain the best The capture rate is defined as the percentage data for comparison of results. of the targeted material that is actually captured from participating households during a It would also be beneficial to audit the organics collection. Organics capture rates can be and recyclables schemes to determine whether determined by collecting refuse and organics the introduction of the scheme has had any from a representative sample of households other impacts, such as reduction in total waste and taking it to an appropriate venue for sorting, produced, and material capture rates. classification and weighing. Although this only Specialist companies are available to conduct provides a snap shot of a limited number of waste audits and it is advised that council properties, it provides useful data on the commission audits by a third party. amount and type of organics being recycled and remaining in the waste stream. This type of 3 Set out rate and analysis needs to be conducted by a participation rate analysis professional. A waste audit can be undertaken in Set out rate is defined as the number of demographically representative areas of the households putting out organics bins for local government area before a new organics collection within a target area, divided by the collection scheme starts. At the end of the trial, total number of households within that area that the audit can be repeated to establish capture have been supplied with an organics bin. rates. This will establish a baseline and may Set out rate monitoring can be undertaken by inform where to target extra communications council, contractor staff, or an external activity in order to ensure good scheme consultant, simply by visiting the target area on understanding and participation. the collection day (immediately prior to normal collection time) and completing a tick list per property of bins presented before applying the formula: No. households recorded as setting out on a given day No. households monitored on that day

3 5 Residents’ feedback Target population

There are various methods of obtaining Monitoring and evaluation activities can be feedback from residents. Focus groups can be tailored to suit a variety of situations, such as conducted to gather opinions, or surveys can covering all participating households, random be undertaken. Although surveys can be households across the trial/service area, or a distributed and returned by post or online, this target area deemed representative. approach does not generally achieve a high response rate. Door to door surveys can help It is important to have good knowledge of the ensure questions are appropriately delivered target population from which the sample is to be and understood, and that a representative taken. Target population characteristics are to demographic of respondents is obtained. be defined in light of those likely to have most influence on the topic under investigation. It is also important to gather feedback from When food organics is being investigated, processors and collection crews to ensure household size and income levels are key operations are suitably evaluated and factors, and the household profile of the target adjustments made as necessary. population must therefore be known so that a representative sample is sampled. For a combined food and garden organics collection service, it is also important to consider the average size of gardens and seasonality impacts.

Demographic profiling can be used to help identify a subset of the target population which is representative of the wider population. Profiling can provide useful insights about the population and give detailed socio-demographic information for categories such as age, gender, social grade, ethnicity, employment status, income levels, housing types and tenure.

Example of survey (Hills Shire Council) Resource Recovery Officers talking to a resident 6 Communications about his recycling bin (Bankstown City Council) evaluation More information on implementation is outlined in Factsheet 13 – Understanding Monitoring It is highly advisable to monitor and evaluate and Evaluation. the effectiveness of all communication methods used. This will help ensure future communication activities benefit from lessons learned regarding ways of targeting different audiences and the ability of different formats to get a message across and stimulate the biggest response.

4 FOOD AND GARDEN ORGANICS BEST PRACTICE COLLECTION MANUAL: PLANNING YOUR SCHEME – FACTSHEET 8 – UNDERSTANDING MONITORING AND EVALUATION Case Studies effective strategy in achieving recycling behaviour change Bankstown Council  Develop a final Contamination Reduction Strategy, including standard Evaluating the ‘Recycle Right!’ and extended contamination Contamination Reduction Strategy management procedures and refined education resources for both houses and Bankstown Council in Sydney’s south-west has units to be implemented across the rest a large, culturally diverse population with a high of the Local Government Area proportion living in high density dwellings. Recycling contamination rates in the dry  Conduct continuous monitoring recycling bin have been extremely high, even and review of the strategy through bin after 20 years of experience with kerbside inspections, waste auditing and recycling collections. Financial penalties have community consultation. been placed upon council for disposal of By the end of the trial period in May 2011, contaminated feedstock supplied to the council officers had inspected and given recycling facility. Responding to these issues, feedback to over 1,400 households, and the council recently instituted a program to ‘offenders’ in each area received feedback over systematically test and evaluate the a three month period, including, two rounds of effectiveness of several resources and personal visits, and a letter warning that bins strategies to improve contamination in the dry could be removed if contamination continues. recycling bin. A similar approach could be Over this time, only four bins were removed. adopted to address contamination in organics collection schemes. The trial stage of the program was successful, but the on-going implementation has seen even The ‘Recycle Right’ Contamination Reduction more dramatic results. Average starting Strategy commenced in 2010 and involved the contamination levels were 40% across the following stages: identified ‘contamination hotspots’ involved in the trial. By the end of each standard  Develop aims and objectives of the contamination management procedure being campaign, aligned with relevant council implemented in an area, the average and contractor policies and strategies contamination rate dropped to 18% and, by the  Undertake a literature review of end of the extended contamination Bankstown waste audit results, management procedure, it dropped to 6%. government reports, studies into recycling, behavioural change research Community consultation and qualitative and other literature on fundamental evaluation were critical elements of the behavioural psychology principles campaign. In total, 700 evaluation forms were completed by households with a 4% response  Design a range of strategies and rate, providing valuable feedback on the resources to trial in both houses and community’s response. In addition, two rounds units, based on the literature review of focus group testing were undertaken to findings inform the design and then redesign of the  Conduct community consultation ‘Recycle Right’ education resources. Results of through a series of focus group sessions the multilingual focus groups indicate the to test the supporting material and community will respond best to the following determine the effectiveness of designs, elements: images and messages for community  Use of multicultural children to members convey messages (aged 8–12 years)  Test and monitor the strategies,  Inclusion of bins in photographic refine resources in several problem areas designs for contamination over a nine month period, and document the results  Use of smiley faces and ‘thumbs up’ symbols  Evaluate the results from the trial strategies and community engagement  Simple and self-explanatory sessions to recommend the most designs

5  Strategies that are informative and Waverly, Randwick and appear cost-effective Woollahra Councils  Cooperative approaches that show council is working with community Using a Planning Framework for Developing an Evaluation Strategy –  Council showing appreciation to ‘The Compost Revolution’ residents for doing the right thing. Based on the results achieved and the The councils of Waverly, Randwick and comprehensive research and evaluation Woollahra in Sydney’s eastern suburbs process that was put in place, the program was received NSW Environmental Trust funding for recognised as: a collaborative project to investigate options for more sustainable management of organic  Winner of the 2011 LGSA waste over a three year period, ending May Excellence in the Environment Awards 2011. (Community Education & Improvement) Randwick and Waverly councils were interested  Winner of the 2012 in exploring the feasibility of home composting Communications Australia Awards (Best and waste avoidance programs as key waste Community Engagement) management strategies, and developed the  Highly commended in the ‘Compost Revolution’ program as a 12-month 2011 Keep Australia Beautiful trial to test a new educational approach. Sustainable Cities Award (Environmental Monitoring and evaluation of the project was Education). built in at the project planning stage, based on the ‘Outcomes Hierarchy Model’, which is an evaluation framework recommended for NSW Environmental Trust grant recipients, which can be accessed using the following link: to Does Your Project Make a Difference. This framework was used as a tool to plan activities and evaluation methods to meet the ‘desired Artwork for the ‘Recycle Right Program’ (Bankstown outcomes’ of the project at three stages: short- City Council) term ‘Immediate’ outcomes; medium-term ‘Intermediate’ outcomes; and long-term The trial provided a Lessons Learnt: ‘Ultimate’ outcomes. ‘Desired outcomes’ were reliable evidence base for the council’s new defined as the changes that they wanted to see approach to dry recycling education and (and measure) in their environment or target contamination management procedures for audience as a result of activities undertaken in both single and multi-unit dwellings, and the the project. process of ongoing monitoring and review will ensure that it can be revised and refined to From the start, the councils identified the maintain its effectiveness. While the system ‘Ultimate’ desired outcome was to reduce was introduced to manage contamination in the greenhouse emissions from waste in the region, dry recycling bin, a similar approach could be including reducing the amount of organic waste adopted for organics recycling services. produced and the amount sent to landfill. To achieve these objectives, the councils identified Further information: several ‘Intermediate’ outcomes to guide the overall structure of the project plan. Once these Detailed information on the contamination were established, ‘Immediate’ outcomes were management strategies and results can be developed by the project team as ways to found in Factsheet 12. Also refer to the Council achieve results. website at: www.bankstown.nsw.gov.au . Individual project activities could then be planned to meet these three levels of outcomes, and suitable evaluation methods were chosen to measure the effectiveness and appropriateness at each stage.

6 FOOD AND GARDEN ORGANICS BEST PRACTICE COLLECTION MANUAL: PLANNING YOUR SCHEME – FACTSHEET 8 – UNDERSTANDING MONITORING AND EVALUATION Because the desired outcomes had been ‘Compost Revolution’ Outcomes Hierarchy clearly identified at the start of the project, the (abridged version) team found developing the comprehensive Project Desired Evaluation monitoring and evaluation plan was very stage outcomes Methods straightforward. Some elements of the Immediate Engage at Trial ‘Outcomes Hierarchy’ for the project are least 300 participation included in the table on the next page. households to rates (about 600 participate households) Lessons Learnt: Detailed forward and start Trial drop-out planning using the ‘Outcomes Hierarchy’ home rates less than framework allowed the team to monitor and compost in 10% revise activities along the way. The results have the trial Workshop and supported decision-making for long-term waste event management in the Sydney Eastern Suburbs participation region, and the effectiveness of this program rates has also led to implementation of the program Develop Phone/email in other council areas of NSW. effective enquiries & education feedback resources and Website visitors workshops to and downloads meet the Number of needs of resources participants distributed at events Intermedia Improve Trial participant te participant surveys during skills and and after knowledge in program composting Workshop Practical workshop being conducted with trial feedback participants (Waverly, Randwick and Woollahra surveys Councils) Comparison with non-participants Raise ‘Food waste Further information: participant tallies’ recorded awareness of by participants http://compostrevolution.com.au/ behaviour Results of Food changes that Diary study lead to Website visitors reduced food and downloads waste Develop a Online quiz model results program to News articles increase and home advertisements composting published rate in region Ultimate Reduce Analysis of organic waste compositional disposed to bin audits landfill before, during and after the trial Reduce Analysis of food household waste production of composting tally food waste system Analysis of 7 ‘Food Diary’ study Reduce Economic and greenhouse environmental emissions modelling from waste analysis undertaken after trial completion