The Limits of Inclusive Leadership Training - An Experiment at Phoenix Group
In late 2021, Phoenix Group, a FTSE 100 financial services firm and long-term partner of MoreThanNow, asked us to evaluate the impact of an Inclusive Leadership Programme. Everyone involved in the project was aware of the limited evidence for the effectiveness of online diversity training, but the team at Phoenix Group were interested in whether bespoke, best-in-class, inclusivity training for leaders would change behaviour.
We’re pleased to present the results of that study, one of the only experimental evaluations of Inclusive Leadership Training ever conducted in a large organisation. We’d like to thank our collaborators from Harvard Kennedy School, Harvard Business School and the University of Exeter for their support. We also want to offer huge credit to the team at Phoenix Group, who conducted this impact evaluation fearlessly and transparently. We hope their pioneering approach - and the results below - will fundamentally change how organisations think about inclusive leadership development.
The Experiment
Our study was designed as a large-scale randomised controlled trial, separating managers into either a control (business as usual) or treatment groups. Treatment group one attended the Inclusive Leadership programme, and treatment group two received the training plus a 6-week series of nudges designed to drive inclusive behaviour in team meetings. We felt this might be an effective way for managers to focus on improving one aspect of their working lives.
The training was offered to 900 senior leaders at the company, based on our earlier randomisation. Our outcome measures were targeted to understand whether the training and interventions changed manager and employee behaviour, and employee sentiment towards belonging in the company, their team, and their relationship with their manager. Our manager behaviour key metrics were focused on key themes from the training, including:
Allyship: One of the key messages from the training was the need for managers to be allies. Joining an employee resource group (ERG) was one of the ways managers were told they could do this. As such, one of our measures was to ask all managers whether they would join an ERG.
Openness to feedback on inclusive team dynamics: Another key training message was the need to solicit and be open to feedback on how to improve inclusivity as a manager. As such, another measure asked all managers to share a short survey among their team to solicit ideas to improve inclusion.
Inclusion advocacy: Finally, we hypothesised that if managers behaviour was more inclusive as a result of the interventions, this would translate to employees being more likely to share a 200-word story about a time they felt belonging in the company to share in marketing materials.
It’s important to note that our behaviours were not intended to capture every possible way in which a manager could behave inclusively – but rather as a ‘proxy’. If we saw changes in the behaviour we could measure, we would predict changes elsewhere. If we did not, we would expect limited changes elsewhere.
Results
Most organisations stop evaluating initiatives like this after a post-training survey. The initial indications from this feedback were extremely positive – 91% of participants rated the training very highly, and no one gave it a below average rating. Additional commentary from managers was highly positive, and came alongside behavioural commitments and suggestions for further roll-out.
The behaviour change results were very different. When examining the data, we were firstly struck that all managers at Phoenix Group had responded well to all the inclusive actions we had asked of them. Many joined ERG groups, and shared surveys with their team to solicit feedback on how to improve inclusivity. However – and this is the important part for our impact evaluation - there was no difference between managers in our treatment and control groups. We saw no evidence that the training made them any more likely to engage in these inclusive behaviours. Instead, simply asking all managers – in a direct and compelling way – to do these things was what made them happen.
This means that our evaluation for the effectiveness of the training on behaviour change is stark: It didn’t change immediate behaviours around allyship, openness and advocacy, and we didn’t see any change in the sentiment of people reporting to those managers over a longer period. If we don’t see changes on these grounds, we don’t think managers are likely to change other relevant behaviours such as hiring more diversity in their team.
What about the nudges?
We would love to be able to say that our nudges were able to optimise the training, but we see no evidence of this either. We saw no effect on behaviour change or team sentiment. The sequence we were hoping was that the training would inspire managers to want to improve belonging in their teams, and the specific actions we were suggesting in team meeting would provide a tangible route to this goal.
In hindsight, we hypothesise that this was still too broad an ask, amid a much broader training curriculum and the three behaviours we had already asked of managers. This lacks the same precision or targeted isolation of our effective treatments in experiments like fairer CV screening at Ericsson, more psychologically safety 121’s at Novartis, or more accurate performance evaluations at a global engineering company.
What next?
Phoenix Group still see high value in the training, but are now realistic about what it can achieve and what it can’t. The programme performed an important role in raising awareness of issues surrounding diversity and inclusion, and increasing knowledge and skills around un/desirable actions they can take as managers. The company’s public commitment to diversity and inclusion through the training also sends an important signal to managers and employees about expectations.
From MoreThanNow’s perspective, this experiment and the results reinforced our evidence that for big cultural change to happen we must be specific about the small everyday behaviours that need to shift – with even more precision than the nudges we designed in this trial. Broad-brush change interventions - like training - set lofty ambitions and may raise intentions. But as behavioural scientists we know there’s a well-documented intention-action gap, and managers may need more everyday support to debias decisions and behaviours. Our work in other organisations has found that when we focus on small, specific behaviours – such as inviting underrepresented candidates to interview, encouraging women to aspire to leadership positions, and managers making unbiased ratings in performance reviews processes – we see more successful behaviour change.
A final note
Naturally, we always hope to see positive behaviour change from our experiments. But to make progress, we also need to diagnose what isn’t working. We use this evidence to again suggest that the resources dedicated to training in DEI are significantly over-weighted, and that more intensive, bespoke training will not necessarily yield better results.
We finally want to highlight one behaviour that we are certain will push us forward – the bravery to evaluate the impact of our work, and to adapt to what we find. To that end, the team at Phoenix Group have and continue to be an exception, a pioneer, and a true leader in the field of DEI. These results are helping them divert resources and explore more effective routes of behaviour change. We hope other organisations are taking note.
This study is the first we will share at our roundtable of DEI Leaders at Harvard Kennedy School on the 19th March 2024, alongside Tamar Hughes at Phoenix Group and peers from organisations including Novartis, AstraZeneca, Thermo Fisher, Merck KGaA, Worley, Ericsson and Regeneron. To make progress, we need even more organisations to step up and commit to further our understanding of DEI science. Get in touch below if you’d like to join this community: