I know, I feel, and I see that what Odd Arts does–works. It changes people’s way of thinking and seeing the world; it opens new opportunities through self-belief and realisation of personal ability; it empowers people to find ways to challenge the things or people that harm us. Proving this is another matter. Of course, we should have to prove we make a difference. There is ever growing scepticism of charity morals and we are at a time of national austerity–proving our impact is necessary and right.
Within its 15-year history, Odd Arts has of course evaluated its work. We have used a lot of techniques: Before and after questionnaires; Case-studies; Recording indicators for change (all the usual suspects). Often showing a journey of real change and impact. However, there have been a few things happen in 2018 that made me really question how authentic this is, resulting in a complete shift in attitude towards data collection, evidence and evaluation.
Did I always believe what our data told us? No. Did it mean we did our job better and served our beneficiaries better? No. Did it help us tick boxes and gain funding? Yes. Our past response: “It’s out of our hands”
This response is not reflective of the values of our organisation.
This is why we have changed.
Telling us one thing, but knowing another…
Let me explain. Recently I took part in Social Value UK’s Impact Management peer learning session. It gave me a little time to sit back and think about the WHAT and the WHY of evaluation and reflect on some of our findings. It made me think in particular about a recent project and the data it produced:
In this project, we worked with men in a community rehabilitation unit (Langley House Trust) delivering workshops around issues relating to resettlement. The male ex-offenders all developed their own interactive forum theatre piece based on their lives and experiences, to re-imagine their future and resolve past issues. The project was everything we hope for as applied theatre practitioners: Meaningful; personal; supportive and with the capacity to be life-changing. However, the questionnaires from the men told me that they had less confidence; fewer employment skills and a reduced ability to work with others. I knew this simply wasn’t true.
Similarly, I have worked on projects where participants on paper show a zero to hero transition, suggesting they had no ability at the beginning of a project, yet on completion were now demonstrating exemplary employment skills and sky-high confidence, when you really know this journey was nowhere near as black and white and was, in fact, far subtler. The self-evaluation questions we have used for years rely on the ability to honestly and effectively reflect on our personal state of being, participants mark themselves between 0 and 6 on their skills in different areas.
But what if, at the beginning of a course, the participant has never reflected on their well-being and confidence, and has no idea what critical thinking skills are? Surely by participating in a project that builds these skills enables them to understand what they mean. Surely part of that journey is the forming the ability to effectively and realistically evaluate their own being and behaviour. Sometimes we complete evaluations at the end of a performance and presentation, when spirits are high, and participants are buzzing… But what about 5 days later when they are alone in their prison cell, would the answers be the same?
What does this tell us about the traditional questionnaires? Maybe they’re meaningless. Maybe they’re not. Definitely they’re unreliable as standalone evidence.
This is why we have changed.
It’s out of our hands… Or is it?
Read Rebecca’s full blog here, which explores some of the key challenges Odd Arts faces when running services that align with their values, measuring impact and working with partner organisations. She explains how they have adapted and developed their programmes and impact measurement processes to make better support those they work with.