Gateshead Big Local

Measuring what Matters Workbook

{name} Big Local, December 2014 Draft workbook

Early Years Evaluation Team

Contact the team on 020 7520 2548 for more information or by email at [email protected] Introduction

Evaluation is where you use information to make judgements on how your programme or project is doing; it’s about assessing the value or worth of what you’re doing. It can help you in three main ways:

Evaluation involves four stages: 1. planning (what will success look like, what do you need to measure to understand how you’re doing?) 2. collecting information (gathering the evidence you need) 3. analysing it (figuring out what the evidence tells you, assessing how you’re doing) 4. using it (to learn, to improve, to share and celebrate your successes)

This “Measuring what Matters” workbook focuses on the first stage of evaluation – that is, planning. In it we share our ten top tips for Big Local areas to plan how they could assess how they’re doing. Plan and commit Plan ahead and see evaluation as an ongoing process to help track your journey as well as your destination. Commit to reviewing, reflecting and being honest about what works and what doesn’t so  that you don’t just prove but also improve as you go along!

Plan ahead so that you can learn as you go Evaluation isn’t just about measuring success, it’s also about learning and helping you improve. Big Local is trying something different. Sometimes it’s about taking risks and trying new ways to tackle old problems. This means we’re all learning together about what works best, so it really helps to build into your plans time to review and reflect on how things are going and what you’re learning.

Remember you can do a lot yourself Sometimes people worry that evaluation is “best left to the experts”. This isn’t necessarily true. It’s important to value the skills and knowledge you already have – listening skills, observation skills, some good story-tellers, and a lot of local knowledge will get you far. Even better if your group has someone who likes to ask difficult questions - “but why?”, “but how?”, and “so what?” are the basis of some of the most interesting evaluation questions!

Planning ahead can save headaches later Planning ahead can save you headaches later because it’s really common when people aren’t sure what information to collect, to then collect either too little information, or too much, or simply not the right information you need to answer your important questions.

Define success clearly  Clearly define success. You can’t measure what you aren’t able to define. Is there anything else we want to consider when we think about what success will mean for us? Define clear changes or results Describe your intended changes clearly and simply. Try to get to the heart of them. Break complicated  changes down and use single, simple change statements to make things easier to measure.

What would you measure in this example?

To make things easier it helps to express the differences you want to make as single, simple statements; to break big changes down; and to try to get to the heart of the change so it’s clear what’s most important.

Activity Activity Change Change Have a story that works Have a story or map that explains the logic of your journey - a place where you’ve laid out clearly your  assumptions about how what you do will contribute to the results or changes you want to achieve.

It’s important to make sure there are no steps missing – sometimes the steps are the things you want to measure.

Looking at your local story map, ask yourself:

(1) Does this tell our story? What would you add to tell a better story?

(2) Does the story ‘work’ – is there a logic to it - or are there steps (changes) missing? Prioritise what matters most Be realistic and prioritise what matters most. You may not be able to collect information about  everything you do as often as you’d like. Prioritising can help you collect less but collect smarter!

You can’t measure everything Big Local is a long-term programme. A lot will be done and hopefully a lot will change. You can’t measure and report on all of it. It helps to prioritise and focus on what are the most important elements of your Big Local story. To decide on what matters most, it usually helps to focus on your goals.

Focus on what most helps you achieve your goals Often when asked to measure their progress, people focus on how much they’ve done, what activities they’ve done. That information is really important, and you may find it’s what a lot of people are interested in – because telling others how many events you’ve held or how many people attended these gives a very tangible way of showing that you’re getting things done. However, you will also (and more importantly) want to know if your activities are moving you closer to your goals, to making the differences you want to make.

Be realistic and try to be proportionate Practical considerations also shape what you measure.

 Some changes aren’t in your direct control or might happen after your work has ended. Focus first on the changes that are in your control.  Not all the changes you make may be equally important.  Some things may simply be too difficult or time-consuming so you need to consider the next best thing that you can realistically measure, remembering that what you do to measure your success shouldn’t get in the way of doing the work you want to do. Ask ‘how will we know’ questions When you’re thinking about assessing whether you’ve made a difference, it helps to think in advance about the signs of the change you could look for. Asking yourself ‘how will we know?’ for each  change really helps.

Asking “how will we know questions” can help you think about the signs you might see, things you could count, or things people could tell you that might help you evidence the difference you’re making. For instance … say one of the big, long- term changes we want to make is that we want to improve our local environment so that the whole community can benefit from it. We have a large park on our doorstep but it is under-used for a variety of reasons. We broke down our big change (improved local environment) into three specific changes that felt most important to us. We said, “We will know we’ve made progress with this change if ...

 Our park is cleaner

 Our park is safer for families to use

 Our park is better used by the whole community

Then we asked, “how will we know these changes are happening? What will we count, measure or observe?” We ended up with the beginnings of a plan as below. Big Change: Improved local environment for the whole community’s benefit Specific changes How will we know? What will we count, measure or observe? What might others tell us? Our park will be cleaner  Level of littering  Level of graffiti  Level of complaints  Level of use of dog litter bins  How clean people feel the space is Be lawyers not scientists – building a case Think of yourselves as lawyers (collecting evidence to make the strongest case you can), rather than scientists (collecting proof).  using different types of evidence or perspectives to explore the same thing.

When we talked about signs of the change, we mentioned you might look for things you can see, things you can count or measure, or things others might tell you. All three could be valid and if you combine one or more you could make a stronger (more credible) case. Likewise you could think of assessing change from three different viewpoints to make a stronger case. Recycle and re-use When it comes to collecting information or evidence there are many toolkits, ideas, questionnaires and forms you could use, and there may also be others collecting information you need which would save  you collecting it yourself.

Is there an easier way? Has someone else done this before? Always check if there’s an easier way to collect the information you need. This way you can avoid reinventing wheels, collecting information that’s already out there somewhere, or duplicating what someone else is doing. Check online or with other organisations working in your area about how they collect information. There are lots of great questionnaires, forms, ideas out there, many of which you can find by searching online. You can often adapt what someone else has tried and tested to collect information you need.

Has someone else got this information or could they collect it for you? Be a recycler! It’s also worth checking what information is already being collected about your community and seeing if you can use it. Your local council, NHS and voluntary organisations and schools may have really useful information you could use. Also, sometimes if you have a vital question you want to ask your community, one of your partners may be willing to add that to their own survey or forms and to share information with you.

Note of caution – do a test run first! If you are using or adapting someone else’s questionnaire or feedback form or using questions others have found helpful, always test them first with a few people from your target audience. What works in one setting or with one group of people, may not work in another. Think about how you’ll use information before you collect it Always think ahead to what you’ll do with information BEFORE you collect it, and try not to collect  information unless you know you will use it.

Think in advance about how you’ll store information you collect Don’t collect personal information if you don’t need it. If you do, make sure you’re able to keep any personal information in line with data protection guidelines.

Think in advance about how you’ll analyse information you collect Remember that once you’ve collected information you have to be able to analyse it and use it. For instance, a survey might seem a great idea until you realise you don’t have time to enter 300 questionnaire responses into a computer, and you’re not sure how to make sense of all the questions people answered in words instead of numbers!

Think in advance about how you’ll use information you collect It doesn’t make sense to collect information you won’t use “just in case it might come in handy one day”. Think carefully about how you want to use information in advance or you could end up collecting information you can’t use because you have too much of it and can’t make sense of it, because you collected it too late to be of use to you, or because it doesn’t really answer your questions. Make the most of being part of a national programme Remember you’re part of a national programme. Local Trust and its national partners can support you  in measuring your progress in a number of ways.

Guidance, toolkits and case studies  The Big Local Annual Review process includes tools to help you review how well you’re doing and to help you assess yourself as a local partnership.

 Check Local Trust’s website or contact the team for a copy of the “Big Local: Checking Your Progress” guide or the equally useful “Checking Your Progress” toolkit which contains lots of ideas and some useful sample documents

 There are lots of Big Local case studies on the Local Trust website to inspire you and give you ideas for how you too could use stories to share what you’re doing and the progress you’re making.

On the horizon  Local Trust plans to set up evaluation workshops to help areas explore more of the “how to” side of things. Training on measuring impact may also be made available in the near future.

 The School of Public Health Research is currently researching the links between community control and health and wellbeing. In time they hope to share useful data and ideas on how areas could look at the impact of Big Local on health in its broadest sense.

 The Early Years Evaluation Team will be producing a database of changes and indicators (signs of change) that areas could visit to look at what others are doing and to get ideas for how to measure their changes over time. Appendix 1. Reminder of our ten top tips for measuring what matters

 Plan and commit to ongoing reflection and review  Define success clearly  Have clear changes and break them down where you can  Have a story that works – that links your activities and your results  Prioritise what matters most  Ask ‘how will we know’ questions  Be lawyers not scientists – build a strong case  Recycle and re-use  Think about how you’ll use information before you collect it  Make the most of being part of a national programme Appendix 2. What a measurement plan might look like for a long-term goal Our specific What we will count, How we will changes measure or observe count it Our big change (what an improved (signs of the change) (methods) local environment means to us) Cleaner open spaces Level of littering Annual litter pick Level of graffiti Council records Level of complaints Level of use of dog litter bins How clean people feel space is Survey : Safer open spaces Level of reported crime Police records Improved Number of accidents in play area Council records local People’s perceptions of crime environment How safe people feel spaces are Survey for the whole More visited/used by Level of use by families Question in community local people Proportions/profile of those using schools parent (eg, age, families) survey Level of bookings for community Council records events and attendance at these The community take Number of volunteers helping Monitoring records better care of the with litter picks and events of volunteers space Level of littering/graffiti (see above) (as above)