Thursday, March 1, 2018
Greater London Authority – Stepping Stones Evaluation
The transition from primary to secondary school is a period of significant change in the lives of young people. Many children embrace and adapt to this change, particularly when they are well supported by parents, families and school. There is, however, a significant minority of students who experience uncertainty, risk, and anxiety around transition, or who are not equipped with the necessary skillset to navigate this change successfully.
The Stepping Stones pilot programme was launched by the GLA for the 2016/17 academic year and was aimed at vulnerable young people considered at risk during the transition from primary to secondary school. The programme was designed to secure improvements in attendance; educational attainment; and behaviour within school, through a range of engagement activities and peer mentoring support.
OPM Group was commissioned by the GLA to evaluate the pilot programme, to provide evidence of what works and why in supporting the transition from primary to secondary school for vulnerable young people. Download our evaluation report and executive summary to learn about the findings from our evaluation.
In addition to our evaluation report, we also produced a toolkit of resources, video case studies and information to help schools implement Stepping Stones in the future. The toolkit can be found here: www.london.gov.uk/stepping-stones
Thursday, December 21, 2017
Evaluation of the Essex County Council’s Family Innovation Fund
Increasing financial pressures on Local Authorities have seen Early Intervention services deprioritised and under-funded. This has left a gap in service for those families who do not qualify for statutory or specialist services but who do need support.
There is now a body of evidence to support the argument that meeting the needs of these children, young people and families at the earliest opportunity will develop the resilience and self-efficacy needed for good outcomes. As a result, there could be a reduction in demand and therefore financial benefit for a range of public services such as Social Care, Education, Health and Police.
The Family Innovation Fund (FIF) was launched in Essex in 2015 and was designed to provide Early Help interventions and support for children, young people and parents/carers with low level additional needs.
OPM Group was commissioned by ECC to evaluate the FIF programme, to provide practice-based evidence for what works in Early Help, and to provide the economic case for further investment.
Download our interactive report here to learn about the findings at the end of this two-year evaluation. Note this document has active functionality – please download via the icon top right:
Monday, August 14, 2017
Demonstrating the impact and value of vision rehabilitation – a Report to the RNIB
Vision rehabilitation services are crucial to ensuring blind and partially sighted people remain as independent as possible. Now, new independent research commissioned by RNIB, with support from the Department of Health, has identified that the cost of providing vision rehabilitation services is dwarfed by the financial benefits.
Independent research by the Office for Public Management (OPM), based on a case study of services provided by Sight for Surrey has shown that the financial benefits of good vision rehabilitation services significantly outweigh the actual costs of delivering this service. In fact in the case study site, over £3.4 million of health and social care costs were avoided, reduced or deferred annually based on a service which cost an estimated £900,000 a year to deliver.
Building on the work of our See, Plan and Provide campaign, we are now working to ensure that commissioners or those making decisions understand the economic value of providing effective vision rehabilitation services and the long term costs avoided, reduced or deferred for the health and social care system.
Wednesday, July 19, 2017
What’s your Corporate Responsibility value? What, how and why to measure impact
How can you know what impact your corporate responsibility activity made? Can you know what’s changed because of what you’re doing?
These questions can lead many down a rabbit hole of thinking programme evaluation and impact measurement is daunting – if not impossible.
As a research consultancy, we’ve heard many thoughts about the limitations of evaluating and measuring programmes and outcomes:
We get it. When corporate responsibility isn’t your day job, or even if it is, the process of designing research tools, talking to the right beneficiaries and analysing your data might be pushed down the long To-Do list.
But, as many participants were likely reminded of at the Heart of the City’s workshop on “Data, Information and Insight: understanding your impact” last week, measuring the value of a programme develops insights that can substantively make a case for your corporate responsibility story: to the Finance Director, the CEO and clients.
This is important. Whether we’ve come to corporate responsibility deliberately or accidentally, it’s hard to ignore the benefits a well-designed programme has on staff, communities and (eco)systems.
But what are these benefits? And, how do we know this? What can be improved?
Understanding your impact supports your organisation in demonstrating how its values and mission are being met and lets your team and organisation improve activities and programmes for the future.
However, while most organisations are great at communicating what they want to do:
“We will reduce poverty in the local area our office operates in by December.”
“We will reduce our carbon footprint by 5% by composting tea bags over 12 months.”
From our experience, there are 3 questions many organisations could improve on asking:
- What will change (for the organisation or communities) by doing what you want to do?
- Why will that change occur?
- What data do we already have that can generate this insight?
But – how can organisations ask these questions differently – or better?
One way is a tool called Theory of Change. This might be the first time you’re hearing of it, or maybe your organisation is already using the tool. But, so we’re on the same page, a recap:
The tool and thinking process to develop it:
- Builds an overview of what you’re trying to achieve and helps organisations think through the necessary steps to realise this long-term outcome;
- Provides a visual summary explaining what activities you’re doing and surfaces what might be useful to measure, and;
- Fills in the gaps between what a programme is meant to do and how it will do it.
Last week, Tarran MacMillan and I were thrilled to work with fellow Heart of the City newcomers during our workshop, “What’s your value: how to know what, how and why to measure impact.”
During the session, we led participants through a Theory of Change exercise using our experience delivering workshops with the tool and evaluating our own corporate responsibility activity that we recently completed for the Migration Museum Project, a UK-based charity.
- Action: what would your organisation like to achieve with its corporate responsibility programme?
- Activities: what does your organisation need to meet this goal?
- Change: what will happen as a result of your organisation’s corporate responsibility activities?
A Theory of Change can be produced at any point, though it’s most useful when developed collaboratively at the outset of a programme and referred to and adapted throughout as the programmes or goals change.
Programme and activity measurement can feel like a task to be done later, but the Theory of Change reminds us that getting started from the beginning helps you track useful data from the beginning that you can use to prove the great work you and your corporate responsibility team are doing.
Have you tried it?
If you’re interested to learn more, contact Dr Peter Welsh.
Tuesday, June 20, 2017
Accelerated Non-Medical Endoscopist Training Programme – Year 1 Evaluation (Report to Health Education England)
The Office for Public Management (OPM) was commissioned by Health Education England (HEE) to conduct an evaluation of the Non-Medical Endoscopist (NME) accelerated training pilot. The NME training pilot aimed to recruit and successfully train 40 NMEs across two cohorts. The first cohort started the programme in late January 2016 and the second cohort started the programme in mid-April 2016.
The evaluation aimed to produce both formative and summative findings about the impact and effectiveness of the training pilot. The evaluation activities consisted of:
- A literature review to understand training outcomes and process learnings from comparable training programmes.
- Interviews with trainees, their supervisors and their mentors from across the two cohorts.
- A survey of Cohort 1 trainees and supervisors and mentors following completion of the NME programme.
- Face-to-face interviews with a sample of patients who received an endoscopy from a NME trainee.
- Analysis of management information and training data.
- Observation at various programme activities, including the second selection day in London, two of the taught study days held in London and Liverpool and a Basic Skills Course.
- Ongoing interviews with programme Faculty members and stakeholders involved in developing and overseeing the programme.
Monday, April 24, 2017
Unleash the creative potential of Social Impact Bonds
In a previous blog based on my latest advisory trip to Japan, I noted that Japan is currently ‘translating’ the SIB model in order for it to be implemented in a way that is appropriate to its specific social, economic, political and cultural context. I have encouraged Japanese colleagues not to simply think that SIBs are and always will be what they currently look like. Instead, they should approach it creatively, making it work better for all. There is a risk that an innovation, such as SIBs, may be abandoned because of disillusionment with early versions of it, which may not have fulfilled the creative potential that may be on offer. Here, I describe four reasons why I think current SIBs have only scratched the surface of what may be possible.
In the UK, we have become rather lazy with our terminology. I often hear people swap ‘commissioner’ for ‘outcome payer’. In doing so, there is a real risk that we limit the way we think about who outcome payers can or should be. Apart from public bodies, who else may be interested in paying for outcomes? What types of outcomes may they be interested in paying for?
If outcome payers are ever only going to be public sector commissioners, then we need to question whether SIBs are indeed channelling ‘new’ or ‘different’ funds. After all, if all outcome payments to investors are ultimately made by public sector commissioners, then the monies will only ever come from direct taxation.
Read any publication or attend any conference on SIBs, and you will head the refrain: “SIBs have high transaction costs”. Again, rather than simply accept this as an immutable fact, I challenge the market to design these costs out of future SIBs. We have good evidence that this is possible, at least for some types of transaction costs. For example, we know that vested commercial interests can cause some intermediary organisations to develop SIBs that are unnecessarily complicated. Similarly our evaluation of the Essex County Council SIB found that when the various players are more concerned about minimising risks to themselves, they can end up with a contract that is too complicated and therefore imposes ongoing costs. As our experience in Essex shows, these can be designed out of a SIB even after it has gone ‘live’.
Who bears the risks?
An attraction of SIBs is that we introduce a new group of stakeholders called ‘social investors’ into the picture, who have higher risk appetites and are socially minded. However, the fact that 7 out of the 10 SIBs in the US have over half of their values guaranteed by philanthropic organisations really causes us to question who is really bearing the risk? Are we attracting the ‘right’ types of investors into the market or are we distorting the market to suit certain types of investors? In addition, there is also evidence that some social investors can try to pass on some of the risks to service providers, for example, by providing part of the capital as a loan rather than as revenue. We must therefore be clear about who bears what risks, and whether these models are true to the ideal of the SIB aspiration.
Is it all about savings?
In a blog I wrote last year, I argued that SIBs do not have to be about savings, and showed different ways of constructing alternatives. SIBs are about social outcomes. To reduce social outcomes to only those that generate financial savings for the public purse is highly limiting. This, again, draws attention to the limitations of thinking of ‘outcome payers’ only as ‘commissioners’. Even amongst commissioners, financial savings do not have to be the only motivator. We need to ask ourselves the question: “Outcomes for whom?” when we design SIBs. If we never ask service users what success looks or feels like to them, then what message are we sending out about SIBs? Whose interests do they serve?
Current SIBs have barely scratched the surface of what may be possible. Rather than allowing them to ossify into what they currently look like, we should challenge ourselves to keep pushing the creative potential of the idea of a SIB. In the process of doing so, we must never lose sight of outcomes and how they can be meaningfully defined.
Dr Chih Hoong Sin, Director, Innovation and Social Investment
Wednesday, February 22, 2017
Independent evaluation of the Essex Multi Systemic Therapy Social Impact Bond
For the past three years OPM has undertaken an independent evaluation of the Essex County Council Multi Systemic Therapy (MST) Social Impact Bond (SIB). The Social Impact Bond delivers MST to children and young people at risk of being taken into the local authority care. The Social Impact Bond has been managed by Children’s Support Services Ltd, a Special Purpose Vehicle, established by Social Finance.
The purpose of OPM’s work has been to evaluate the potential added value that can be achieved through local authority and other commissioners using Social Impact Bonds as a mechanism for financing the delivery of new services. Specifically, whether the use of a SIB impacted on the implementation of MST and whether significant value was added to either outcomes or performance.
The evaluation highlights many instances where the use of the SIB has been seen to add value to systems and processes and indirectly to overall performance. It also draws attention to some additional costs and complexities which may result from operating through a SIB and how these can be mitigated. Drawing on a variety of quantitative and qualitative evidence and insights, OPM have now summarised their findings and used this to inform a set of recommendations applicable to organisations that may be considering developing SIBs of their own.
In addition, OPM have drawn on the evaluation findings to develop a new guide on ‘Top Tips for developing and implementing a Social Impact Bond’ – an interactive document for commissioners, providers, funders and managers. While summarising the Essex experience and including examples from Essex to illustrate specific points, this also brings in themes from the wider evidence base, with the aim of distilling lessons that have wider applicability.
The Essex SIB was the first local authority commissioned SIB to be established in the UK and the experience and findings reflect its innovative nature. It was launched in 2013 and will be operational for five years, concluding in 2018. Two teams from Action for Children have delivered an MST intervention to approximately 260 young people to date resulting an in an approximately 80% rate of successful care diversion. MST is an evidence-based programme that seeks to improve parenting and rebuild positive family relationships, enabling families to manage future crisis situations themselves.
Friday, January 27, 2017
Evaluation report – Learning into practice project
OPM was commissioned to produce an evaluation report on the Learning into Practice Project (LiPP) which was funded under DfE’s Innovation Programme and ran by NSPCC and SCIE.
The LiPP was testing a proof of concept – aiming to establish what is needed on an ongoing and sustainable basis to improve the quality and use of Serious Case Reviews (SCR) in England. The LiPP consisted of four main workstreams:
- developing a mechanism for collating and producing accessible information on practice issues and causes from SCRs
- the establishment of a strategic Alliance of national strategic and leadership bodies to consider and implement improvement work, from a national perspective, as a result of SCR findings
- supporting commissioning and conduct of reviews through a set of Quality Markers
- improving lead reviewer expertise through a series of masterclasses
Our evaluation was asked to explore:
- stakeholder perceptions of the potential for the new mechanisms being developed and tested to achieve improvements in the quality and use of SCRs in the future, and in turn, better outcomes for children and families
- what might be needed for these mechanisms to become sustainable and implemented on an ongoing basis
The evaluation involved 63 qualitative interviews with those involved in LiPP activities; and an online survey aimed at non-participants in the LiPP activities to explore wider views on the proposals. 126 people completed this. Alongside the external evaluation, the project team conducted an internal evaluation of the LiPP, focussing on describing the mechanisms being tested, and the emerging learning from these.
Tuesday, September 20, 2016
Reflections on evaluating the Wandsworth Care Coordination Centre
We recently completed our evaluation of the Wandsworth Care Coordination Centre, based at Royal Trinity Hospice, which is a pilot service providing coordination and care to people approaching end of life.
Overall we found the pilot to have been a success, with positive impacts for people at the end of life, their families and carers, as well as for other professionals working in the local end of life care system. Rather than repeating all our detailed findings here, I wanted to reflect on a few points which I personally found particularly interesting about this project and what the evaluation discovered.
The holistic view
Firstly, one of the real strengths of the Centre is how, through coordinating people’s end of life care support, it has an overview of people’s experiences and needs, and brings together all the services that help them in their last months and days. We found that this was positive for people and their families, as well as representing a more effective way of working for other health and social care professionals involved in end of life care. The crucial point here is that the Centre is able to take a holistic view of a person at the end of life and their care, which is invaluable in the complex end of life care landscape. The importance of this holistic view in end of life care is now widely recognised – not least as a key quality principle in NICE’s guidance for end of life care.
Secondly, what is true for the individual at the end of life, in terms of taking a holistic view, we also found to be true for the local end of life care system as a whole. It’s important to remember that the success of the Centre is linked to how well it interacts with other parts of Wandsworth’s local system, from district nurses and GPs, via Royal Trinity Hospice’s other services, to physiotherapists and specialised CNSs, relaying information back and forward, and coordinating the efforts and dedication of all the other professionals involved. But through coordinating all this activity, the Centre also gains a kind of holistic perspective on the local system overall; and in occupying this central, coordinating position, the Centre is able to identify systemic blocks in local processes, and try to have these resolved. In this way, it was able to improve pathways – ultimately bettering care provision for people. This represents an added benefit of having a coordinating function in a system as complex as end of life care.
Contributing to an under-researched field
The last point that really piqued my interest was methodological. We knew that the role of coordination in end of life care had been under-researched and under-evaluated. Particularly with regards to economic evaluation approaches for end of life care and care coordination, the evidence base is still rather slim (with some exceptions, such as the NAO’s study, and the economic evaluation of EPaCCS).
In our own small-scale, local economic evaluation of the Centre, we mainly worked with the Centre’s existing data, but nonetheless were able to demonstrate monetisable benefits for the Centre’s work. A lot of this hinges on assigning value to the reduction in avoidable admissions, with that reduction recognised as something that is notoriously difficult to quantify, and attribution in a system with multiple interventions is tricky. Nonetheless, we were able to run through scenarios for what kind of monetisable benefits the Centre may be achieving, suggesting that there is value in coordinating functions for end of life care (something we also found in a previous study we undertook for Macmillan).
There’s clearly still a lot more research to be done, but it’s been great to contribute to the development of knowledge in this field through our evaluation of the Centre. Of course, no two models are exactly the same, with local context a huge factor influencing variations in service provision; but hopefully the lessons we identified from Wandsworth can help other commissioners and service providers in considering whether such a model would work for them elsewhere.
If you want to talk to us about how we can support you to evaluate services in end of life care, please contact Chih Hoong Sin.
Monday, September 19, 2016
Our evaluation finds end of life coordination centre a success
Our evaluation looking at the impact of a new End of Life Care Coordination Centre based at Royal Trinity Hospice has found that patients using this centre are less likely to be admitted to hospital or to die in hospital at the end of their lives.
The was designed to address confusion among patients and their families about who to contact for help and support, due to the range of organisations involved in caring for someone approaching the end of their life.
It comprises a 7 day nurse-led coordination team and helpline for patients, families and professionals based at Trinity; a dedicated St George’s End of Life Community Nurse; and a team of Marie Curie Health and Personal Care Assistants who can offer specialised hands-on care at home for people with any terminal illness. It was commissioned by Wandsworth CCG as a two year pilot in February 2015.
We found the centre resulted in avoided admissions and faster discharge from hospital, which could reduce deaths in hospital. The centre was also found to free up clinical time for local healthcare professionals. The evaluation calculated these benefits could amount to potential savings of almost £350,000 to the NHS in the first year of the pilot, although these have to be offset against the cost of delivering the model.
The evaluation also found the centre improved the quality of end of life care through patients and families feeling supported at home and feeling reassured that they will be looked after. The quality of care arranged and provided by the centre was also deemed to be improved.
Dallas Pounds, Chief Executive Officer at Trinity, said, “Most people want to be cared for at home at the end of their lives rather than spending their final days in hospital. We now have evidence to show that the Wandsworth End of Life Care Coordination Centre is helping this become a reality for more people.
We hope other hospices and healthcare providers are able to draw from this evaluation so more people can achieve their final wishes at the end of life.”
The final evaluation report can be found here.