News and Comment

Mickey Mouse and bits of string: valuing consultation responses

Tuesday 8 April 2014


Consultation responses come in many formats:  random newspaper cuttings, photographs of people’s back gardens, bits of string, scrapings from bird baths, 500 page reports, single word responses, small plastic bags containing earth. But of all the different types of response that we receive, none have the power to stir up as much debate, reflection on our analysis and reporting processes as campaign responses – a subject we have written about previously.

Campaign (or organised) responses are responses submitted by individuals on behalf of a particular campaign. Usually these responses feature standardised text stating the group’s position on the relevant policy or plan. Sometimes this text is a single statement and sometimes it includes many statements with adjacent tick-boxes so that respondents can indicate which of several points they agree with. Many campaign groups suggest that individuals personalise these messages. In one recent consultation, we received thousands of responses from something like 30 different campaign groups. Many were on pre-printed postcards and in some cases the statements from one campaign group differed from those in another by one bullet point only or even just by the name of a town. Many people did personalise their response, adding a few words or several additional paragraphs.

As my colleague Beth noted in an earlier post, as engagement specialists we have a responsibility to maintain an ethical conscience, to carry out good quality research and to be reflective about the strengths and limitations of our work. Given this, there is a pertinent question practitioners must consider in regard to campaign responses to consultations. Namely: should we treat pre-printed campaign responses differently to responses from individuals who have taken the time to send in a long and detailed response?

Is a consultation a vote?

Some people see consultations as something akin to a public vote, in which the most frequently raised points responses carry the most weight. But if it is the numbers that count and a public consultation is something like a vote, then does the format or origin of responses matter? Surely if it’s a vote and a response comes from a real person then it counts (we do get responses from Mickey Mouse and some of his friends and we do discount these). So if you get 15,000 campaign responses, these are 15,000 votes for or against something or another. The justification for segregating out campaign responses here is hard to see.

A consultation should provide a channel through which anyone with an interest in a set of proposals can raise the issues that they think need to be taken into account in the development of the proposals. This makes them closer to qualitative than quantitative exercises – and it’s why we encourage the use of open questions rather than tick boxes. One person could make a relevant and important comment on a campaign response that is not discussed in any other response: you wouldn’t (or shouldn’t) dismiss it on the grounds that it came on a campaign postcard from just one person, any more than you would (or should) give 1500 times more weight to a single issue commented on in 1500 submissions, campaign or individual. Unlike a survey or a vote, in a consultation it should be the issue raised, the point made that is important, not the numbers.

When is a campaign response not a campaign response?

But let’s say that you want to treat campaign responses differently to ‘original’ responses – and for often very valid reasons, clients do want to do this. The problem then is identifying these campaign responses. If the campaign response has been personalised – and many of them are – does it still count as a campaign response? How much personalisation is needed before a campaign response becomes an ‘original’ response? If you spot a certain number of phrases in an individual letter that look very similar to the wording on a campaign group’s website, do you count this as a campaign response or an individual response?

Making judgements

This is about making judgements – being reflective about the strengths and limitations of a particular approach in a particular situation – and choosing the one that is most fair to respondents and most useful for those who will use the consultation findings. I worry sometimes that arguing for the segregation of campaign responses is really about making a judgement about individuals who use this way of responding to consultations. That they’re not really invested in the issues (if they were, they would have taken the time to write a ‘proper’ response); or that they haven’t understood sometimes complex proposals (as if a consultation response was in some way a test of this understanding and you could dismiss responses that you don’t think pass that test); or that the respondent has in some way been pressured by the campaign group (as if respondents lacked their own mind). This is dangerous ground to tread.

To my mind, a good consultation should seek to be as fair and accessible to people who might want to respond as it is useful to the client.  On any engagement project, we’re working for the client: but doing this well means working for participants too. This includes being clear about your approach to analysis – for example, providing information about how you intend to treat campaign responses, so that people can choose the format in which to respond. Changing the rules midway through, when it becomes clear that a campaign is underway, is unfair on respondents and not good practise.

All consultation presents challenges and these differ from project to project: how do you analyse a bit of string?