A look under the surface of a public consultation: Part 3/5 – Analysis and coding
Analysis of responses is a crucial stage of any consultation process, but all the more so when the subject of the consultation is potentially controversial.
Consultations such as that commissioned by the Human Fertilisation and Embryology Authority (HFEA) on new IVF-based techniques to avoid mitochondrial disease are likely to attract a significant number of responses, some concerned with the implications for scientific research, others contributing to a wider ethical debate. The duty of the analysis team is to accurately capture the essence of each of these comments, enabling decision makers and others to obtain an overview of respondents’ views.
When we brief analysts at Dialogue by Design, we make sure they are aware just how important their task is. We remind them that the data that they are organising consists of people’s views, and that these people have taken the time and effort to respond to the consultation because the subject means something to them. As independent contractors, we like to think of ourselves as working not only for our client, but also on behalf of the respondents. This notion informs our approach to the analysis process, which is characterised by transparency and rigours auditing.
Without the right tools it would be very difficult and time consuming to analyse the sometimes intimidating volumes of data that big consultations can generate. The HFEA consultation on mitochondria replacement attracted almost 2,000 responses, many of which answered most or all of the seven consultation questions. Also, as discussed in our previous blog discussing planning and design, respondents used different channels to respond, and indeed quite a few sent handwritten letters or response forms. We are lucky to have systems that are purpose-built for dealing with consultation responses, and these allowed all data to be ready for analysis in one central database soon after the close of the consultation.
Tools alone, however, are no guarantee for a thorough and efficient analysis process. They facilitate the analysis, but at the heart of the process are the analysts who structure and organise the responses. And that starts with the content. A ‘coding framework’ is designed for any analysis process – short phrases summarising reoccurring themes in responses, such as ‘small quantity of mitochondrial DNA’ or ‘impact on family relationships’. The framework is developed on the basis of what respondents say, so that the codes reflect the views expressed rather than what is expected beforehand. The coding framework evolves throughout the analysis process, shaped by respondents’ comments and coordinated by our lead analyst.
Equipped with the analysis database and the coding framework, the analysts read each individual response to each consultation question and ensure that codes are assigned to every part – if one phrase contains several arguments we will typically assign multiple codes to it. This is the crucial feature of the analysis: it organises a seemingly endless volume of qualitative data into manageable little sections. This not only allows our writers to summarise the responses into a report (more about reporting in the next blog in this series),but also enables decision makers to access the detail of specific arguments made in relation to particular themes and issues.
For high-profile consultations it’s even more critical to ensure that the right issues are identified throughout the analysis, and that our codes aren’t short on detail or clarity. With this in mind we provided the HFEA with a live connection to our analysis database, so that they could review how our analysts assigned codes to responses. This mechanism ensures we receive helpful and timely feedback on our work, also meaning that respondents’ comments are presented to decision makers in as close to real time as possible.
This is the third blog in a five-part series which analyses how Dialogue by Design, (part of the OPM Group), consults and reports on contentious subjects, paying special attention to a recent consultation conducted on behalf of the Human Fertilisation and Embryology Authority (HFEA).
In the next installment in the series – The next step is to write a summary report once the analysis process is completed, which gives an account of the views expressed in response to the consultation. We will look at the specifics and challenges of the report writing process in detail.
‘A look under the surface of a public consultation’ blog series: