Measuring the Impact of Health and Wellbeing Coaches

Author: admin

  • From “how much?” to “what if?” and back again

    From “how much?” to “what if?” and back again

    When we want to measure the effects of an intervention or service, it’s tempting to start by asking "how much?" — jumping straight to measuring outcomes without thinking about what else besides the intervention may influence our results. The causal model of health and wellbeing coaching we have developed through countless discussions with expert practitioners allows us to a different set of questions: those that begin with "what if?". (For more details on the model, see here.)

    Much like with a pharmaceutical intervention, where the precise effect of administering a drug depends on diverse factors such as dosage, adherence, interactions with other medications, and so on, so too is the effect of health and wellbeing coaching influenced by a wide range of factors. Our causal model captures these different factors and their relationships, showing, for example, that the gender of the coachee (an aspect of the coachee position) can directly affect both the coaching itself and the outcomes we’re interested in. People differ in how they respond to coaching, and also in how confident and adept they are in managing their own health and wellbeing; the model proposes that these differences are partially determined, to a greater or lesser degree, by gender.

    Having identified these effects, we can now ask a "what-if?" question: What if all of the coachees were of the same gender, keeping everything else the same? How would that change the outcomes? As long as there are enough coachees in our dataset who have the same values for all model inputs except for their gender, we can estimate the effect of gender by comparing the difference in average outcomes for these two groups. This difference is equivalent to the change in average outcomes we would see if everyone was of the same gender.

    The value of asking "what if?"

    Asking the what-if? question about gender gives us two different types of insight:

    1. Knowing the estimated effect of gender on outcomes allows us to get a better estimate of the direct effect of health and wellbeing coaching by identifying and removing a source of bias. Previously, we didn’t know how much of the outcomes we observe is due to the coaching and how much of it is due to gender effects; now we can remove gender effects from the equation and get a less biased estimate of the coaching effects. Asking "what if?" helps us answer the "how much?".

    2. The estimated effect of gender may suggest ways in which a health and wellbeing coaching service can be improved. For example, let’s imagine we find that, all else being equal, the effect of gender on outcomes is negligible (i.e. people of different genders are equally likely to experience positive outcomes), but more women are being referred for or attend coaching than men. We can use the insight about equality of outcomes to encourage more referrals for men, and to encourage more men to attend coaching. Being able to demonstrate that coaching works just as well for people like them as for anyone else can help break down some of the barriers that lead people to assume that coaching is "not for me".

    The same approach works for any input variable in our model (i.e. any aspect of the coachee position, the coach position, or the coaching modality). What if every coaching session was face-to-face rather than remote? What if every coachee was coached by someone who was experienced in working with people with their specific needs? Answering each of these questions gives us more insight into the direct effects of coaching and may point to opportunities to improve health and wellbeing coaching services. With data from a small number of coachees, we might only be able to identify some of the what-if? effects, getting closer to an unbiased estimate of the coaching effect but still leaving many unknowns. As our dataset gets larger, however, we can resolve more and more of the what-if? scenarios.

    Changing the question

    It’s easy to reduce impact measurement to a simple sizing up of outcomes, but there’s much more to be learnt by thinking about the various causes of those outcomes, and what would happen if those causes were shaped differently. Drawing on the knowledge of expert practitioners to build a causal model of our intervention can allow us to explore the different effects that lead to real-life outcomes, using what-if? questions to estimate the size of those effects.

    Asking "what if?" also helps us improve health and wellbeing coaching services by identifying levers we can pull and what their effects are likely to be — for example if we made changes to the way the coaching is delivered, or targeted different underserved populations for coaching support.

    What are the what-if questions you would want to ask about factors affecting your own work?

  • First test PCNs now live!

    The first PCNs in our testing cohort have been trained and are now testing the framework with real coachees! This is a real milestone for the project, when everyone’s contributions are brought together and we get to see how well things work in practice.

    We’re still expecting some bumps in the road – the stabilisers of our bicycle remain firmly attached – but the feedback from our testing cohort will bring us another step closer to understanding how to measure the impact of Health and Wellbeing Coaches in a rigorous way that encapsulates the key things that coaches and coachees care about.

    The next milestone is the month 3 technical check-in, when we take an early look at the data being collected and discuss early feedback with the testers. You’ll be able to read all about these early findings here!

    There are still a couple of spots remaining on the testing cohort – if you would like your PCN to take part then please contact us for more information.

  • Feedback from the online consultation

    Thank you to everyone who submitted their thoughts on the draft model and framework as part of the recent online consultation. There were 9 responses received from 8 participants, all of whom are Health and Wellbeing Coaches. This post summarises the thoughts and suggestions shared and our responses to them.

    Strong cases were put forward for adding additional measures to the framework to measure the following:

    • The presence of someone else in the coaching session along with the coachee (for example a carer, family member, or translator) – it was agreed that this can significantly alter the coaching dynamic
    • The use of contracting – establishing a contract is considered a core part of coaching, so it was agreed that the framework should indicate whether this was used or not

    We were not convinced that in the following cases the extra insight gained outweighed the potential cost or impact on the coaching relationship:

    • Coachee level of education / cognitive capacity – it was agreed that this would be very difficult to measure without impacting on the coaching relationship, and furthermore its effect on outcomes is fully or largely mediated by variables that are already measured in the Coachee Position section of the framework
    • If you didn’t access coaching what would you do, have you accessed coaching before, reason for DNA – all potentially interesting questions, but we were not sure that they have a big enough impact on outcomes to merit inclusion
    • The framework assumes that HWBCs are operating within the scope of practice for their role, as described in the PCN DES; it is not an appropriate tool for monitoring or addressing issues related to out-of-scope practice, which should be taken up with your ICB’s Quality and Safety team

    The following proposed changes were not considered persuasive enough at this point:

    • Specialist support is already covered in the measure ‘Involvement of other professionals in delivering the coaching’
    • It was acknowledged that a lot of people don’t know what health and wellbeing coaching is, but the feeling was that the best way of addressing this is to keep using the term until it becomes normal, rather than using something more generic and less informative
    • We are inclined to keep the measure around coaching methods and techniques, even though it might be difficult for some coaches to answer consistently – a frequent criticism of existing research into the impact of health and wellbeing coaching is that the research doesn’t define what the coaching consists of, so it is not known exactly what is being assessed for its impact on outcomes
    • We are inclined to stick with the language around barriers to good health and wellbeing, but monitor the utility of this measure during testing, as the proposed alternatives had their own issues around clarity and interpretability
    • We understand that data gathering should ideally occur in the normal course of the coaching conversation, in which case anonymous or remote feedback wouldn’t be possible – this will be covered as part of the training in how to use the framework, including how to obtain informed consent from the coachee to use their data

    We agreed with the following points:

    • The full framework as it currently stands doesn’t make sense for one-off coaching sessions – we will develop a cut-down version of the framework for one-off sessions
    • There are a handful of measures that require revised wording to make them clearer or use more appropriate language

    As we get into testing the framework with real coachees, we will likely discover that some of these calls need revisiting, and that there are other issues not raised during the consultation that need addressing. This is the purpose of testing! We’re looking forward to continuing the conversation as testing progresses – if there are any further thoughts you’d like to share please do contact us through our contact form, or (even better) post to the Health and Wellbeing Coaches forum on Future NHS.

    There are still a couple of spots left for NHS-funded employers of Health and Wellbeing Coaches who would like to be involved in testing the framework with their coachees, so please do get in touch using our contact form if interested. Onward to testing!