I’ve always been really captivated by the world of evaluation and feel energised whenever a conversation at works turns to monitoring, evaluation, and learning. Don’t worry, I know I’m in the minority of people on one end of a spectrum, with many more on the end that die a bit inside when they hear those three words (or one of the many many acronyms associated with them).
I think this is really interesting. Having worked in the community and voluntary sector for the past 10 years on both the side of community worker and funder I know the tensions and issues that evaluation brings up in the context of funding, so I understand the palpable discontent when MEL enters the chat.
But I want to put that to the side for now, because I think the issues with evaluation run deeper than funders looking at qualitative outcomes beyond outputs, proportionate evaluation for grant size, or capacity building organisations to have the tools to measure change.
I think these conversations, while important, can get in the way of the issues lying beneath them, limiting us to surface level debates about what the world of learning and evaluation could look like.
Here are some of the thoughts I’ve been having which feel like the foundations – the things we need to be discussing if we have any hope of moving beyond evaluation being a necessary evil, towards something that feels exciting and intrinsically connected to the purpose of the work.
We need to stop using evaluation and learning interchangeably.
Lumping learning and evaluation into the same sentence (and worse, tacking monitoring on there and giving it an acronym) does a huge disservice to learning.
I think of evaluation a bit like a fitness watch. If your goal is to get fitter, and you start going for runs, you might at some point progress and decide to get a fitness watch to track your progress. You learn how to record all your stats and metrics, and one day you go for a run and forget your watch – if you didn’t record the run and put it on Strava, did you even run? We all know the answer is yes, and obviously you still got the benefits of the run, but if you’ve been here before you know that a small part of you feels like it didn’t really count.
It’s extremely valuable to train with clear metrics, allowing you to follow a long-term plan and adjust in real-time based on feedback. If you want to run an official marathon, it’s necessary to have measurable and comparable metrics to measure how well you’re doing against others, and gain access to certain competitions. But if evaluation is the watch, learning is the run. You don’t need the watch to reach your goal of getting fitter. What you need is to run.
In the goal of positive social change, asking questions about what we should start, change, keep doing, or do differently in practice, is something everyone needs to be doing. Conversations that explore and understand these reflections in more detail will uncover patterns that point to not only what did or didn’t work, but in what context, for who, and because of what elements. Even if it’s not formally structured, measured, and recorded, that learning process has happened, and it will change how you’re working.
Evaluation provides the structure to test theories and approaches, looks at patterns from a zoomed-out perspective, and a way to communicate your learnings to a range of audiences using commonly agreed metrics. It isn’t a necessary component to learning though, and importantly, it’s not something everyone involved in social change needs to be doing.
We need to be explicit when we’re talking about a structured process to measure and demonstrate impact, and when we’re talking about reflective learning to do better. Coming back to the analogy, we don’t want to be skipping the run because we know we’ll be slow and we don’t like the metrics, or because our watch died so we can’t track it.
The risk of bringing monitoring and evaluation into every conversation about learning is that the pressure/lack of tools/lack of interest leads to not engaging in a learning process at all.
The way we measure our work now doesn’t fit complexity.
If we’re committed to changing systems (which we are), we need to think first about how systems work and whether our tools and processes are appropriate. Let’s go back to the fitness watch – it can tell you when and where you started your run, how far you went, your heart rate, and how fast you went, both overall and broken down into segments. That’s pretty cool and that’s a lot of facts to help you compare one training session to the next. What it doesn’t really tell you is why those numbers were what they were, and what conditions might have influenced the outcomes of your run. You may know your typical 5k time and you can see you went slower than normal on your usual route, but does that mean you are getting less fit or does it mean that you had a family in front of you with a pram and you didn’t want to be rude and push them out of the way, so you ended up running half a mile behind them really slowly?
The systems we work in are like going for a run outside – there are different, emergent conditions that come up in the world around you, not to mention in the complex system of your body, which mean that even if you apply the same approach every time, the outcome can change in ways you can’t predict or control.
If you want to understand how you are doing in your goal of getting fitter (or enabling more children to be active), using tools like a fitness watch (or attendance metrics) that only tell you what happened aren’t going to give you a sufficient story. Taking notes of the factors and circumstances that you think may have contributed to specific outcomes will give you a fuller picture of how something is changing, allowing you to test out new approaches and hopefully do a better job of creating the change you want to see.
It’s messy. We may never be able to fully understand the impact of our work.
For me this is the hardest, but one of the most important ideas to come to terms with. Understandably, we want to clearly be able to point to the change we’ve been desperate to see and say that because of this specific thing I did, that happened. Unfortunately, when complexity is involved, that’s potentially an impossible goal to chase. The conditions that affect systems outcomes are multiple, evolving, interconnected, and unpredictable. To return to our runners for the final time, even as an elite athlete with access to the world’s best nutritionists, coaches, and facilities, a win isn’t predictable or guaranteed. Sometimes you have the worst training camp of your life and you’re sick and you win the race, and sometimes you create the perfect conditions both in your body and environmentally and you lose. And worst of all, you don’t really know why.
That might lead some people to feel like what’s the point in trying to make sense of why things happen at all then, but if you think about what we do know from observing patterns, there are general conditions that are shown to be more likely to lead to success. In running, this might be eating enough carbs around training, and in systems change this might be collaboration. Focusing on understanding and then creating the conditions for success without getting caught up in trying to replicate processes and controlling for every detail allows us to be genuinely curious and make space for new ways of doing things to emerge.
Part of the challenge here is letting go of our ego and trusting that if we are creating the conditions we can see contribute to the change we are trying to make, that we are playing our part in the system to change the outcome.
At Miova, we’re thinking about how to have conversations about learning and evaluation in the context of systems change in the physical activity sector that feel different. I don’t think one day everyone will get an excited buzz thinking about learning in practice, but I’m really excited by the idea of making that happen. As the systems we work in become increasingly complex, learning and reflective practice will make the difference to whether we make the changes we want to see.
