News and Comment

Making the case for flexibility in standardised measures of social impact

Tuesday 12 January 2016

By:

Making the case for flexibility in standardised measures of social impact

Shared measurement and standardisation is a hot topic in the field of monitoring and evaluation, particularly in the third sector. Although there are several challenges to overcome, a flexible approach to shared measurement that is driven by those closest to the participants of social programmes could carry many benefits.

So why would we want to standardise measures in the first place? Here are a few suggestions:

  • Standardising measurement can support learning between programmes that are working towards similar aims, providing a shared language and supporting collaboration to maximise the impact on programme participants.
  • Through developing a shared language and understanding of what works and why, decision makers are also better equipped to make comparisons between different options.
  • As a result, the sector can develop a stronger, more unified voice in discussions with policy makers and funders.
  • As the old adage goes, “what gets measured gets done” – if the same things are being measured by multiple organisations it is likely that more time, energy, and resources will be invested in them.
  • In the current context of frequent media criticism, standardised indicators of impact and success could help the third sector counter or pre-empt attacks with robust evidence and a common language.

This all sounds great, but standardisation also raises a number of obvious challenges. For example – with so many indicators to choose from, to what extent can different individuals and organisations agree what to measure? And in striving to agree shared measurements, is there a risk that we revert back to system-driven thinking where what matters most to participants gets diluted or overlooked?

I would argue that this depends on the type of questions and what their starting point is. One option is for standardisation to start at the level of overarching principles rather than specific metrics, and from the perspective of those experiencing the intervention or support. For example, this question from Saville Kushner seems to me to be a good starting point:

“How well does the program serve, respect, and respond to these participants’ needs, hopes, and dreams in this place?”

There are a number of reasons why I like this question: the first thing you have to do to answer it is find out what is important to participants; it provides flexibility to respond to the specific context; and it lends itself to be incorporated into service delivery rather than as an add-on. Also, the question gives me energy – I want to know the answers (there will of course be more than one) and explore what any similarities and differences might tell us. A question that gives people energy is often a great starting point because it indicates that we are measuring what matters.

In our own experience of supporting organisations to agree shared measures, particularly in person-centred health and social care, we have found flexibility to be a key success factor. Working together to devise a set of agreed standard measures for a sector or type of initiative can bring significant benefits but they often work best as a suite of standard measures to choose from. Furthermore, even where there are standard measures in place, we would usually suggest looking beyond them, and supplementing with bespoke measurements that capture the unique impacts of a particular programme in addition to those which are comparable across organisations.

Unfortunately, measurement has become more combative the higher it has moved up the political agenda. Those delivering programmes can feel frustrated if measures are imposed as conditions of funding and do not align with their own knowledge about what is relevant. And decision makers can be unresponsive to new ideas about measurements, particularly if they are less likely to result in straightforward and comparable quantitative data. For all the talk of person-centred approaches to service delivery and measurement, these do not always have traction where systems are entrenched in traditional thinking and priority is given to reducing costs in the short-term. I am generalising, but it’s certainly all too easy to fall into a trap of measuring what’s easy instead of what matters.

This is why there is a need for the third sector to get ahead of the game and set the agenda for shared measurement. If those providing social programmes can work with participants and each other to agree a suite of flexible standard measures, those measures are more likely to remain relevant to the participants themselves while meeting the varying needs of different stakeholders. In a challenging space of dwindling resources and increasing pressures, it is understandable that the energy and appetite for taking this forward might be low. Yet, the fact that so many discussions are taking place, and that these discussions are happening at the national, sector, and grass-roots levels, suggests that there is already much to build on and scope to increase collaboration.

My aim in writing this is to gather my own thoughts and continue the conversation. I’d love to hear your own reflections…because none of us will come up with the answers on our own.

In the spirit of dialogue I’d like to add that the thoughts laid out in this blog post have been influenced by the following events, individuals, articles and organisations: