The Experimenter Series: Audit and Risk with Nafa Ben Haddej
The Experimenter Series is a collection of six interviews with people pioneering scientific experiments in large organisations. This time, we speak to Nafa Ben Haddej, VP in Behavioural Risk Audit at NatWest about risk and experimentation.
This interview series sheds a light on experimental research in the workplace, exploring how different people from diverse companies make it happen. In this instalment, the brilliant Nafa Ben Haddej shares his take, drawing on experience from both the FCA and NatWest Group. My interview with Nafa follows on from conversations with Rosario Saud at Ericsson, Chris Rider at Novartis, Professor Maria Guadalupe at INSEAD and Brett Skinner At Nationwide Building Society.
“If you don’t run an experiment, you run a bigger risk.”
— Nafa Ben Haddej
Gus: Thank you so much for speaking to me Nafa! I really appreciate it. I already know a bit about the behavioural risk team at NatWest, and I think you are doing some great things. Can you tell me a bit about yourself and your entrance to experiments?
Nafa: Happy to help! I am also interested in what everyone else is saying. To start with some background, I have a Masters in Behavioural and Economic Science from Warwick. Before that, I did a degree in Economics as well. For a while, I worked at the FCA with Alexandra Chesterfield, who I now also work with at NatWest. From my time there in the public sector, I have realised experimentation is perceived very differently in the private sector. I am definitely not referring to NatWest here, but generally speaking, there is a widespread misconception of its value in the business world.
G: Yes, I’ve heard that, but it seems counterintuitive! Why is the public sector paving the way for experimentation and driving innovation? Why is there such a large discrepancy with the private sector trailing behind?
“Yet, all these supposed challenges connect to a misunderstanding of what experimentation means and what it can achieve.”
N: Budget is an important factor. It is difficult to convince stakeholders that there is value in experimental research. There is also the factor of perceived risk. For example, if you have multiple treatments you are testing on customers, the stakeholders might say: “I’m discriminating against some of my customers, and if so, maybe they will get bad outcomes. I don't necessarily want to take that risk.”. Time also plays a big factor. Especially if they are trying to roll out a new product or strategy, the last thing people want to do is stop and test things.
Yet, all these supposed challenges connect to a misunderstanding of what experimentation means and what it can achieve. It is sometimes confused with pre/post testing, but then we’re not really talking about causality anymore. This gives the impression to an organisation that they are doing experimental research when this is not the case. In this vein, the lack of a dedicated team within a company doing this kind of research might result in it not happening, or happening legitimately, in the workplace. If there is no one really joining all the dots together of all the different teams, marketing, data scientists etc. the process gets increasingly difficult.
G: What would you say to those who argue it takes too much time, or attracts too much risk to conduct experiments?
N: The first thing I would say is that if you don’t do it, you run a bigger risk. Secondly, we can make the process run quicker by just signing it off now. [laughs]. But really! The longest part of the experimentation is often debating whether we need an experiment [laughs again].
I’d also argue that the standardisation of the process saves time in the long run. The design of an experiment will challenge you to clearly define goals and remits of power of those that oversee the data and testing. This quickens the process, as a lot of time is often lost in terms of structural things like aligning the teams.
G: Does it maybe help with convincing people if you have quantified proof? Some data to back up your claims.
N: Absolutely. When something has been rolled out, it becomes the new status quo in a sense. The convincing to do the testing has to happen beforehand. It can be quite difficult to get the sign-off, but once you get the data, it forms a compelling argument. Our team at NatWest differentiates itself as is in audit, so the focus is not necessarily on designing and trying for solutions. But rather we are the ones checking, analysing what we have and finding problems. However, experimentation still has a lot of value for us as well. It is also for us a convincing mechanism, a benchmark for success of what we think could have been an alternative. This would put us in the front seat of evidence-based auditing where we can quantify the room for improvement and this is quite a novel space.
G: That must be quite different from your work at the FCA right? Also, the transparency of the experiments done in the private sector versus the public sector I imagine must be quite different. How have you experienced this?
N: The public sector has accountability towards the public, they need to explain how the taxpayer’s money is being spent. Additionally, the public sector carries a lot of thought leadership. If the public expresses concern about why a certain policy or change is implemented, this transparency allows for them to see the evidence behind the policy. Which again, makes a strong argument. Also, scale is an important factor here. Regulatory policy is targeting tens of millions of customers so getting that wrong has very heavy consequences.
On the flip side, companies often feel they need to keep their results secret; they often do not want to share what they are doing. Because if their competitors know what they’re doing, they feel they potentially lose their edge. In most cases, I see this as a mistake: Publishing behavioural research is a brilliant way to show you are leading the way, on a topic that matters to the world, whether risk, internal culture, or optimising customer outcomes.
G: I completely understand the hesitancy to publish, but it is such a shame. And it is very telling to hear a risk professional saying they’re often being overcautious! I am sure many companies have great insights but that they fear the results could be misinterpreted. I guess that’s what stops them?
N: Absolutely. The people that work in these different institutions are also often different. For example, in government institutions, there are many academics, a lot of PhD students, which increases their drive to publish. In a sense, it becomes a kind of research body. Private companies do not have the same commitment to thought leadership, as they need to have some return on investment. So, if I'm bringing a PhD student to work on something, it should get me some return. In public institutions, the focus is on tackling bigger problems in the future.
G: That makes sense. Hopefully, as more and more companies commit to thought leadership and contributing to real-world problems, we’ll see things changing. How do you think experimentation in the workplace will look like in the future?
N: This is a really interesting question. The way I view it is that at some point experimental economics and data science will merge and move to real-time testing. Meaning that the interventions will be adaptive. This would be managed by robots or machine learning engines that are self-optimising. So, testing, improving, testing, improving, but also individually targeted. Meaning, that we have to move from the one size fits all. In research, this sentiment has been echoed on nudging, that we should move away from the one-size-fits-all nudge. This individual, targeting real-time is where it will go.
G: Oh, that's really cool. Do you think this will again be led by the public sector?
“when you look at innovation, the private sector is leading the way. But if you look at the whole population, the number of companies doing it, very few are doing that.”
N: We must define the terms more, so when we spoke about experimentation, it was in relation to the private sector being a bit more conservative and sceptical. That's for the spread of experiments in terms of overall volume. But when you look at innovation, the private sector is leading the way. But if you look at the whole population, the number of companies doing it, very few are doing that. But when looking at capability, tech giants like Facebook, they're doing tremendous work when it comes to testing. That kind of speed then becomes an advantage for the private sector. So, the private sector is leading on innovation. But the public sector uses more experimentation in terms of volume.
G: Is there anything else you would like to mention, or you feel I have forgotten to ask about?
N: Yes actually. In my opinion, it is the responsibility of a behavioural scientist to educate the private sector on what behavioural science means. They often start with finding a data scientist, which is a good start, but that is not really an experimental scientist. That is not a social scientist, not a psychologist. So, I think it is very important to create awareness of the behavioural science role and what it actually entails. And we can start advocating for behavioural science teams where you find a complementary set comprised of a behavioural scientist, a data scientist, a psychologist, a designer, etc …
G: Well, that is exactly what we try to do at MoreThanNow and with this series! I completely agree about this sense of responsibility behavioural scientists should have. Well, thank you very much for all your great insights!
“When something has been rolled out, it becomes the new status quo in a sense. The convincing to do the testing, thus, has to happen beforehand.”
A massive thank you to the incredible Nafa Ben Haddej, who was so kind to speak to me and share his takes on experiments in the workplace. It was interesting to learn that the lack of dedicated teams forms such an obstacle, and how widespread the misinterpretation of experiments is in the private sector.