Tuesday, June 7, 2016

New evaluation of STEM Learning Triple Science Support Programme

We have just completed a two-year evaluation project for STEM Learning (a national body providing schools with continuous professional development in science, technology, engineering, and maths), looking into the impact of the Triple Science Support Programme (TSSP) in 2014-2016. The TSSP supports schools to develop their triple science provision at GCSE level. In our evaluation we spoke to teachers and delivery staff, and found that the TSSP has helped schools make considerable improvements in their triple science provision. We found the project had a range of positive impacts, including raising subject teachers’ confidence; improving science departments’ capacity and capability to teach triple science; and outcomes for students such as improved motivation, and progress and attainment in triple science. Download the full report here.

Thursday, May 26, 2016

Our work with the Royal College of Nursing to showcase the costs and benefits of nurse-led innovation

With funding from the Burdett Trust for Nursing, we have been working in partnership with the Royal College of Nursing (RCN) to build nursing capability in economic assessment and to help nurses demonstrate the value of nurse-led innovation in practice. In April we held a breakfast event jointly with the RCN and Hospice UK, Demonstrating the value of nurse-led innovations in End of Life Care, which shines a light on nurses working in hospice and care home settings. The full video of the event – including presentations from Dr. Chih Hoong Sin (OPM), Dr. Ann McMahon (RCN) and Antonia Bunnin (Hospice UK), not to mention case studies from practitioners themselves – can be accessed via here.

For any questions regarding this partnership or our economic evaluation work, please contact Chih Hoong on csin@opm.co.uk

Tuesday, May 17, 2016

Evaluation of the Reducing Social Isolation and Loneliness Grant Programme

Social isolation and loneliness in older people is a widespread issue that has gained much attention in recent years. We know that being isolated and lonely can impact on a person’s quality of life and lead to more intensive use of health and social care services.  

In Manchester the three Clinical Commissioning Groups provided grant funding targeted to reduce social isolation and loneliness amongst Manchester residents aged 50+. Grants were awarded to voluntary sector organisations to deliver 27 projects across the city. The Programme ran from September 2014 until March 2016 and was managed by Manchester Community Central (Macc).

OPM was commissioned to evaluate the Grant Programme. The evaluation sought to demonstrate outcomes and provide evidence around ‘what works and why’.

This presentation was delivered at the final Programme celebration event attended by representatives from the CCGs, other North West CCGs, Manchester City Council, Macc, local research organisations, plus VCS leads and volunteers from across the city. It presents the headline findings from the evaluation and showcases two projects in depth. Our final evaluation report will be available in the coming months.

Click here to download the presentation slides.

Tuesday, May 17, 2016

Evaluation of the Reducing Social Isolation and Loneliness Grant Programme

Social isolation and loneliness in older people is a widespread issue that has gained much attention in recent years. We know that being isolated and lonely can impact on a person’s quality of life and lead to more intensive use of health and social care services.

In Manchester the three Clinical Commissioning Groups provided grant funding targeted to reduce social isolation and loneliness amongst Manchester residents aged 50+. Grants were awarded to voluntary sector organisations to deliver 27 projects across the city. The Programme ran from September 2014 until March 2016 and was managed by Manchester Community Central (Macc).

OPM was commissioned to evaluated the Grant Programme. The evaluation sought to demonstrate outcomes and provide evidence around ‘what works and why’.

This presentation was delivered at the final Programme celebration event attended by representatives from the CCGs, other North West CCGs, Manchester City Council, Macc, local research organisations, plus VCS leads and volunteers from across the city. It presents the headline findings from the evaluation and showcases two projects in depth. Our final evaluation report will be available in the coming months.

Tuesday, January 12, 2016

Making the case for flexibility in standardised measures of social impact

Making the case for flexibility in standardised measures of social impact

Shared measurement and standardisation is a hot topic in the field of monitoring and evaluation, particularly in the third sector. Although there are several challenges to overcome, a flexible approach to shared measurement that is driven by those closest to the participants of social programmes could carry many benefits.

So why would we want to standardise measures in the first place? Here are a few suggestions:

This all sounds great, but standardisation also raises a number of obvious challenges. For example – with so many indicators to choose from, to what extent can different individuals and organisations agree what to measure? And in striving to agree shared measurements, is there a risk that we revert back to system-driven thinking where what matters most to participants gets diluted or overlooked?

I would argue that this depends on the type of questions and what their starting point is. One option is for standardisation to start at the level of overarching principles rather than specific metrics, and from the perspective of those experiencing the intervention or support. For example, this question from Saville Kushner seems to me to be a good starting point:

“How well does the program serve, respect, and respond to these participants’ needs, hopes, and dreams in this place?”

There are a number of reasons why I like this question: the first thing you have to do to answer it is find out what is important to participants; it provides flexibility to respond to the specific context; and it lends itself to be incorporated into service delivery rather than as an add-on. Also, the question gives me energy – I want to know the answers (there will of course be more than one) and explore what any similarities and differences might tell us. A question that gives people energy is often a great starting point because it indicates that we are measuring what matters.

In our own experience of supporting organisations to agree shared measures, particularly in person-centred health and social care, we have found flexibility to be a key success factor. Working together to devise a set of agreed standard measures for a sector or type of initiative can bring significant benefits but they often work best as a suite of standard measures to choose from. Furthermore, even where there are standard measures in place, we would usually suggest looking beyond them, and supplementing with bespoke measurements that capture the unique impacts of a particular programme in addition to those which are comparable across organisations.

Unfortunately, measurement has become more combative the higher it has moved up the political agenda. Those delivering programmes can feel frustrated if measures are imposed as conditions of funding and do not align with their own knowledge about what is relevant. And decision makers can be unresponsive to new ideas about measurements, particularly if they are less likely to result in straightforward and comparable quantitative data. For all the talk of person-centred approaches to service delivery and measurement, these do not always have traction where systems are entrenched in traditional thinking and priority is given to reducing costs in the short-term. I am generalising, but it’s certainly all too easy to fall into a trap of measuring what’s easy instead of what matters.

This is why there is a need for the third sector to get ahead of the game and set the agenda for shared measurement. If those providing social programmes can work with participants and each other to agree a suite of flexible standard measures, those measures are more likely to remain relevant to the participants themselves while meeting the varying needs of different stakeholders. In a challenging space of dwindling resources and increasing pressures, it is understandable that the energy and appetite for taking this forward might be low. Yet, the fact that so many discussions are taking place, and that these discussions are happening at the national, sector, and grass-roots levels, suggests that there is already much to build on and scope to increase collaboration.

My aim in writing this is to gather my own thoughts and continue the conversation. I’d love to hear your own reflections…because none of us will come up with the answers on our own.

In the spirit of dialogue I’d like to add that the thoughts laid out in this blog post have been influenced by the following events, individuals, articles and organisations:

Wednesday, October 14, 2015

Evaluation of the Multi-Systemic Therapy Social Impact Bond

Evaluation of the Essex Multi-Systemic Therapy Social Impact Bond

Essex County Council (ECC) Family Operations Service currently provides access to Multi-Systemic Therapy (MST) to young people who meet the referral criteria. This service is aimed at supporting young people and their families where there is a risk of a young person entering care (which also includes young people remanded into custody), and has the intention of keeping the young person within the family home whenever it is safe to do so. This service is being funded via a Social Impact Bond (SIB) and is being delivered by an external provider.

In 2013 OPM were commissioned to deliver a three-year independent evaluation of the MST SIB, using funding from Central Government.

The evaluation will generate:

The evaluation involves capturing both qualitative and quantitative data. This is the first of two interim reports and presents the findings from the first eighteen months of evaluation activities. The evaluation will run until March 2016, culminating in a final summative report.

Wednesday, October 7, 2015

‘Quality of death’ – national success vs local variation

Most of us are familiar with the concept of ‘quality of life’ when thinking about a neighbourhood, city, or country. ‘Quality of death’, on the other hand, isn’t something we regularly talk about. However, a recent report has been in the news this week for doing just that.

The latest Quality of Death Index has been released comparing the quality of provision around care for those at the end of life in 80 different countries.

Happily, if that’s the right word, the Index ranks the UK as highest in the world for its quality of death, citing comprehensive national policies, extensive integration of palliative care into the NHS, and strong hospices as reasons behind this.

Certainly in our own work we have come across many examples of good practice, and innovative and effective approaches. Most recently at OPM we were able to visit Trinity Hospice, as part of our current evaluation of the Wandsworth Care Coordination Centre, to see what ‘quality of death’ can really mean in practice.

From our research we’ve found that key ingredients to this include:

However, although it’s great news the UK ranks top in the Quality of Death Index, the authors acknowledge that there’s clearly still room for improvement, which is reflected in our own research, as well as recent case reports from the Parliamentary Health Service Ombudsman.

For instance, the nature of the service landscape for end of life care, which the Index doesn’t take into account, is something we have found to be an important factor in influencing how a person spends the last months and weeks of their life. Often, many different services from across health and social care can be involved in a person’s last days. Where these work in a joined-up manner it can make a huge positive impact, but where this is not the case, people can experience delays, poor communication, and inadequate care.

All too often there can also be huge variability between and even within different areas for how easily people can access end of life care, the nature and quality of care they receive, what location they spend their last weeks in, and, not least, whether they have to pay anything for it.

Perhaps what we need is something similar to the Quality of Death Index for within the UK, to make clear the wide range of experiences people are having in different parts of the country. Of course, this idea would still need a lot more development to make sure it supports improvement rather than just singling out areas for criticism. But done successfully, this would enable us to better identify and share the excellent practice that we know is happening in many areas already – and ultimately improve our quality of death even further.

Monday, September 21, 2015

OPM continues social impact bonds knowledge sharing relationship with Japanese universities

Earlier this month we were delighted to welcome an SIB research delegation led by Meiji University back to OPM.

The visit is the latest development in the partnership supporting a 5 year empirical study funded by the Japanese Government into how social impact investments, especially SIBs, affect governments, social service providers, service users, and the standard of social services in the UK – further evidence of the interest internationally in the progression of the UK social impact bonds market since the world’s first was implemented in Peterborough Prison 5 years ago.

The delegation first visited OPM last November to hear about our experiences evaluating the Essex County Council SIB and ‘Peninsula LIST’ project, and continuing the relationship in April OPM’s Director of Business Development Dr Chih Hoong Sin spoke at the 2015 Social Investing and Corporate Social Responsibility (CSR) Forum, held at Meiji University in Tokyo – presenting his observations of the nature of the ‘first wave’ of UK SIBs and the lessons to be learned from the world’s most developed market.

OPM’s expertise, it is hoped, will contribute to the launch of the 1st Japanese social impact bond.

This latest meeting was particularly timely. At the time of writing the UK still accounts for the largest number of SIBs globally (31), having been the first to pioneer the pay-for-performance vehicle that leverages private funding to finance public services five years ago. In addition, Social Finance had recently announced details of the first UK social impact bonds to perform above expectations and deliver outcomes sufficient to return investor capital earlier than expected. 

Yet the international backdrop is more mixed.

The Riker’s Island SIB, which aimed to reduce recidivism among 16 to 18-year-olds who entered New York City’s Rikers prison by at least 10% had been terminated due to failing to achieve the agreed targets, while a new report from the Brookings Institute (Chih Hoong Sin is referenced as a study participant on page 52 and OPM’s evaluation of the Essex Family Therapy SIB features on page 84) this summer called for increased transparency and knowledge sharing on the potential and limitations of impact bonds to move this agenda – how to better ensure the achievement of outcomes for vulnerable populations – forward.

In this context we were in a position to update our Japanese colleagues on the progress of our evaluation of the ‘Essex SIB’ – the first in the world to be commissioned by a local authority – and the ‘Peninsula LIST Project’, that aimed to use an SIB as vehicle to commission public services across 4 local authorities in the South West of England.

______________________

If you would like to find out more about OPM’s evaluations of the ‘Essex SIB’ and ‘Peninsula LIST’ projects, please contact Chih Hoong Sin, Director of Business Development at CSin@opm.co.uk or on 0207 239 7877.

Tuesday, August 11, 2015

OPM research into SEND pathfinder programme published by the DfE

The final report from the evaluation of the special educational needs and disabilities (SEND) pathfinder programme, featuring qualitative research conducted by OPM, has been published by the Department for Education.

The DfE commissioned a consortium led by SQW to undertake the evaluation. The team drew together a wide range of complementary experience as shown in the diagram below:

 

SEND Pathfinder programme diagram

 

The evaluation has been on-going since 2011 and has described and analysed the work done to develop new approaches to deliver Education, Health and Care (EHC) plans across 31 local authority areas, and the resultant impact on families.

The report considers:

The report contains data gathered through:

Friday, August 7, 2015

Evaluation Framework led by OPM’s Heather Heathfield selected to highlight impact made by UK academic Computer Science Research in new report

The Project Review and Objective Evaluation (PROBE) toolkit led by OPM’s Dr Heather Heathfield has been selected by the UK Computing Research Committee (UKCRC), Council of Professors and Heads of Computing (CPHC) and British Computer Society (BCS) Academy as a case study for a new report highlighting the impact made by UK academic Computer Science Research over the period 2008 – 2013.

Designed initially to identify the barriers that prevented the NHS from adopting electronic health records, PROBE is based on a six step evaluation process, in which decision makers would be asked to agree on why an evaluation was needed; when; what and how to evaluate; analyse and report and finally to assess recommendations and decide on a course of action. Since its revised publication in 2001, PROBE has sparked the NHS to look at evaluation as a key strand of major IT projects, and has become a central part of NHS information strategy and policy.

Overview of the Evaluation Process

In 2012, NHS Connecting for Health’s Head of Patient and Public Partnerships praised Dr Heathfield’s work as having “greatly improved understanding of ICT projects” by focusing analysis on the important questions, and providing more information on the tools and techniques available for effective evaluation.

The PROBE methodology is still widely used in the UK. For example, it was employed by the National Patient Safety Agency to evaluate systems designed to improve the safety of blood transfusions.

The full interview with Dr Heathfield can be found in Chapter 6 of ‘The Impact made by UK Academic Computer Science Research’ report.