The Experimenter Series: The Explainer

 

The Experimenter Series is a collection of interviews with people pioneering experimental research in organisations. But what do we really mean by an experiment? I asked MoreThanNow’s founder, James Elfer, for a quick explainer.

The Experimenter Series by Guusje Lindemann

Below is the transcription if you’d rather read, with links to further resources at the end.

 

The Transcript

“If there was a competition for a word of the year, I think ‘experiment’ would be up there. Which is strange because people seem to use the term in so many different ways. If you asked a hundred people in organisations, you’d get a hundred different answers. Gus, our behavioural researcher at MoreThanNow, asked me for my two cents as part of our new Experimenter Series, so I’m going to give it a go. ”
— James Elfer, Founder, MoreThanNow

First, I think everyone should be included in the world of experiments, so I’ll start by saying that they are simply tools for learning; for building knowledge. My caveat - that we’re going to go on to explore here - is that is that you need to make sure that the tool you’re using is appropriate for the question at hand.

So, if you are thinking about how to change your own behaviour, it’s okay if you just try something and see what happens. As long as you’re clear about what you’re trying to achieve, you set some time constraints for yourself, and you’re open and curious about what you’re going to find, I’m fine with calling that a personal experiment. Why not? It’s a simple tool for a simple question. And promoting it’s use it good for people and the organisations they work for… 

“If, however, you’re designing and testing ways to influence the behaviour of hundreds or thousands of people, as Leaders and HR teams do in so many organisations, that ‘try and see’ method is not the right tool at all.”

If, however, you’re designing and testing ways to influence the behaviour of hundreds or thousands of people, as Leaders and HR teams do in so many organisations, that ‘try and see’ method is not the right tool. It’s like using a hand trowel to try to dig a trench; an endless task with no end. 

To actually learn about your impact on behaviour at that scale, you need your experiment to unlock a principle that sounds simple but is really hard to pin down: Causality. Now that’s my word of the year. Did the thing you do – your intervention – influence… cause… the change in behaviour you wanted?  That’s different from correlation - not my word of the year [laughs] – which just means two variables like employee engagement and performance have a positive relationship, but you don’t really know which one is influencing the other. Does high-performance cause engagement or does high engagement cause performance? Without an experiment, who knows? And that’s without even starting to ask whether some intervention you’ve designed actually caused a positive impact on either variable. 

We’ve got to be able to do better than that. 

“To actually learn about your impact on behaviour at that scale, you need your experiment to unlock a principle that sounds simple but is very hard to pin down: Causality.”

To start thinking about causal research, you can take your cue from experiments in medical science, which I’m sure you’ll agree, have proved their worth recently! So it is worth dissecting some of these principles: I’m going to skip right past the bit where you set yourself a really great, precise research question, and assume you have a clear intervention, something you want to do, and a clear outcome, something you want to achieve. That’s not easy by the way, and you can read about why what we Thinking Small (breaking big, thorny challenges into smaller answerable questions) is so important on our website. But let’s say that bit is done and you’re ready for evaluation stage.

First, you need to select a group of employees that represents your whole organisation. That’s your sample. And within that sample, you need to compare one group, who receive your intervention, with another that don’t. That’s your control and treatment group.

So far, so good.

Your control and treatment group need to be the same in all respects except one is being treated with your intervention and the other is not. When you start thinking about that, you might assume that it’s very tricky. I remember hearing about it for the first time and it sounding impossible – how on earth do you isolate the effect of a Covid-19 vaccine, say, from all sorts of differences in the way people in your treatment and control groups might wash their hands or socially distance? In the same way, wouldn’t two groups of employees who received your intervention be exposed to all sorts of different events and personal circumstances over the course of an experiment? Won’t that ruin your results?  

Well, yes. But you can deal with this in a devilishly simple way. When you divide your sample into your treatment and control groups, you need to make sure that that allocation is random. In a similar way that you can assume that a coin tossed enough times will reveal results that are half head and half tails, so you can also assume that the diverse behaviours and characteristics within your sample, when randomised, have been divided evenly across into your treatment and control groups. So, in effect, they are, on average, the same.  

The randomisation process is one reason why the size of the sample you’re experimenting with is important. The other comes at the end, when you’re detecting differences between the treatment and control groups. You want any effect to be statistically significant, which really means a high degree of confidence that any differences is actually caused by your intervention, rather than a variation that might occur by chance. For that, you need to choose your sample size carefully.

When put together, these principles become the rather grandly named Randomised Controlled Experiment – the term you’ll see us repeat over and over again at MoreThanNow, and for good reason – it’s the gold standard of behaviour change evaluation! If you are approaching this method for the first time, it might sound unusual. But when organisations don’t follow it, they don’t really understand the impact of their work. Because they haven’t isolated their interventions from other changes that are happening in the organisation. They can’t unpick correlation from causation. So all that talk about return on investment – as serious and corporate as it sounds - is a bit woolly; it doesn’t hold up to scrutiny. And because of that, we have organisations out there that are preforming worse than they could be. And there’s real potential of causing harm to employees inadvertently, as we’ve seen time and time again in wellbeing, mental health, diversity and inclusion.  

If we care – really care – about those outcomes, how can we not care more about the causal impact of our attempts to change them? There are millions of people working in large organisations, and I think they better. 

“The pandemic seems to have catapulted us in the mainstream, with more and more large companies wanting to build their capability to design and run experiments”
— James Elfer

MoreThanNow have been doing this type of research for years – and yes, it was very niche when we started [laughs again] - but the pandemic seems to have catapulted us in the mainstream, with more and more large companies wanting to build their capability to design and run experiments.

Really, that’s why I’m talking to you now. And why we’ve we’re celebrating some our pioneering partners, friends and collaborators that have been with us along the way, from institutions like INSEAD and the London Busines School, to organisations like Novartis, Ericsson, NatWest and Nationwide Building Society. This isn’t fringe anymore.

I’d love you to read their interviews as a next step. And I hope that you’re so inspired by their work, that you’ll considering changing your approach, and running your own experiments in the months ahead. When it comes to imagining a new Future of Work, we can’t return to guesswork or overconfidence of the past. We need as many people as possible, especially in large organisations where we can run our most robust experiments, testing and learning a better way. And I hope in some way, this will encourage you to join that community. Thanks for listening.

Further reading

Well, we hope we have piqued your interest! This series does not need to be the end of your relationship with experimentation, it could even be the beginning! However, it is important to do experiments the right way if you want to make an impact. So, if you truly want to dive in – do not hesitate to reach out to the team.

Intrigued? Here are some other resources that might be of interest:

For a basic introduction (to send to your boss 🙂):

For a more detailed description (to start your learning):

Educational options (to get real serious):

 
 
Guusje Lindemann