Harnessing data for regulatory impact: The power of evidence and evaluation

In our recent webinar reviewing the key findings of the Data in Regulation report, panellist Dr Sian Hughes answers questions from panel facilitator Naomi Nicholson on the challenges and benefits of working with multiple large datasets and the importance of using data as part of evidence-based decision making in regulation. 

Note: The following content is an edited transcript from the Data in Regulation webinar.

Naomi Nicholson: The Equality and Human Rights Commission (EHRC) brings together and analyses data from a number of really large datasets and has brought about significant changes as a result. People can read a little bit more about that in the report. In the meantime, can you tell us a bit more about this, and why you think working with those big datasets can be so impactful, Sian?

Dr Sian Hughes: I'll firstly tell you a little bit about how the Commission works and our background. We regulate an absolutely enormous space. So we enforce the Equality Act, our country’s human rights compliance, public sector equality duty, and discrimination and unlawful acts can happen in literally any sphere of life to anybody with a protected characteristic. And we all share at least three protected characteristics, so this affects everybody in every sector, and it’s an absolutely enormous space.

We are not regulating a market where the outcomes are obvious and partially endogenous; we’re a moral regulator. So there’s no automatic system where we actually understand the parameters and can put that into context so we can act. This is why we developed our measurement framework back in 2017. This allows us to put all the different dimensions of equality and human rights in Great Britain into one robust, coherent framework which reflects the multi-dimensional nature of equality and human rights.

So we've got our six domains of life – education, health, justice and personal security, participation, work and living standards. For each one of those domains, we specify measures or indicators that we think are important outcomes or experiences for people in the country. Each of those is stratified by each category of a protected characteristic, and then that's analysed over time.

A further complication is that there's a fair degree of correlation between a lot of these factors. We also have a reasonable suspicion of there being quite significant underlying variables. So, this is like inverting a very complex matrix and trying to pick out the patterns. The only way we can actually assess the state of equality and human rights is by using large datasets with all of those different dimensions in decent quality, over a good timeframe and combining them into one source. Analytically, this is a bit of a headache but in terms of impact, it's the only way we get to see the whole picture.

The only way we can actually assess the state of equality and human rights is by using large datasets with all of those different dimensions in decent quality, over a good timeframe and combining them into one source. Analytically, this is a bit of a headache but in terms of impact, it's the only way we get to see the whole picture.

By doing this, we summarise our findings by a protected characteristic, and by domain. And we look at what is actually happening for these groups in these areas. And it's the only way we can shine a light on some of the disadvantage and discrimination faced by people in this country.

We do have our alternative sources of information. Every time we take on a legal case, that's a data point, but relying on a lot of probabilities so we have to develop a decent critical mass of evidence to be able to say, ‘this is a systematic issue’ rather than ‘this is an isolated issue’. That's a risk we can't take. Especially when we have at our fingertips, a growing improving data landscape with an awful lot of data linkage and infrastructure.

This is really where we start to see those systematic issues and the bigger picture in the space that we regulate.

One of the things that we do need to be careful of with large datasets is that they can be really variable in quality. So, we know that there is much more utilisation of admin data. Indeed, I think there's a move at the ONS to use this data in population estimates and instead of doing a 10-year census, but we're relying on the interest of the data owners in making sure that it is a minimum standard for statistical work and for research work, as well as for the operational purpose that it was originally designed for.

The more we use those datasets though, the more we find out what works, the more we find out what doesn't and what needs to improve. And as part of our data and evidence strategy at EHRC, we're exploring how we can use our influence and our levers to bring about data improvement. We want to improve our use of data even more and move away from this ‘big-bang assessment’ approach where we combine all these datasets and just analyse them once every five years and use it more as a rolling surveillance methodology so we're automatically bringing these datasets into a sort of smoke detector process where we look for warning signs and emerging issues.

We want to improve our use of data even more and move away from this ‘big-bang assessment’ approach where we combine all these datasets and just analyse them once every five years and use it more as a rolling surveillance methodology so we're automatically bringing these datasets into a sort of smoke detector process where we look for warning signs and emerging issues.

(You can read about this in the report case study). To do that, again, the only thing we can do robustly is by using these large datasets. They are pretty instrumental in how we regulate, we wouldn't be able to do our job without them.

Naomi Nicholson: So fascinating. One thing I really loved about this research was hearing all of the variety around how regulators are using data. You've described it so well. You've got that incredibly complex space of what's happening every day to everybody in the country, and you've created this framework and using data in all of those different ways, multi dimensionality, looking at how that all fits together. It's fantastic to feel the importance of that data quality, as you say. When we spoke interview, you talked a lot about the importance of using data as part of evidence-based decision making. How would you describe the benefits?

Dr Sian Hughes: It's a tricky question to answer in some ways, because to us, as analysts, it should be completely obvious, right? That you don't make a decision without data or evidence. These data-driven insights are really of paramount importance for us when we're trying to exercise our regulatory powers strategically and effectively. Without data, we're essentially just hoping that we get it right. With data, we're making an informed decision to act in a particular way.

These data-driven insights are really of paramount importance for us when we're trying to exercise our regulatory powers strategically and effectively. Without data, we're essentially just hoping that we get it right. With data, we're making an informed decision to act in a particular way.

We’ve spoken about our measurement framework that we use to conduct equality and human rights monitoring, that's just one source of data that feeds into our overall understanding of the state of play. That's like the macro picture that helps us decide on our direction for our strategic plan for three years at a time. As well as that, we have a micro picture. This is our daily decision-making: Do we act on this piece of intelligence that we've received? How important is this piece of intelligence? What is the impact of us doing something? Should we do something?

So, all of these micro decisions have to be fed by data, otherwise we have quite a high probability of missing the mark. We can't act on every single piece of intelligence, so we use this for prioritisation and we have a regulatory triage process so we triangulate multiple sources of information. We look at media correspondence, legal developments, monitoring information from our equality advice support service, stakeholder intelligence, etc, and we try and build this cross-sectional picture of what is important today, or what's important this week, and what we have to take action on.

These micro decisions that we make frequently, sometimes they have to be made without a really comprehensive picture because we don't have six months to go away and do a research study. Sometimes you have to act quickly. Without that reassurance of robust research, the only way that we can do that is to have our data safety net. So we have our regulatory model, we have our triage evidence, and we have our monitor. We have all of this in our back pocket, ready to bring out on a daily basis whenever we need to make decisions.

The other point I want to talk about when using data in decision making is having the right monitoring systems and evaluations in your projects and programme so you're not just feeding data in; you are creating data. So if you act and it is successful, that is one piece of evidence. The more you do that, the more you create a real sense of what works in the space that you work in and know what does good look like? What was the impact of our action? Data is essential in answering all these questions. Without data, we couldn't be confident in our decisions. And when we act confidently, we act decisively. It builds credibility, it builds effectiveness. It gives us better value for money and ultimately enables us to do better, to regulate better, to create better outcomes and better impact.

I was really struck by your report finding, that data is not an end in itself. I completely agree with that. Data is a way of life, it's a way of working. That golden thread of evidence-based decision making throughout everything we do is really important. And I'll be championing it for the rest of my career.

Audience question: What’s the best lesson that you've picked up along the way?

Dr Sian Hughes: Sometimes any decision is better than no decision. If you don't have all of the information you need, look at the risk, look at the potential outcomes and just act. That then becomes a data point you can use for your future decision making. Don't be afraid to take risks.

Further information

• To watch the full panel discussion, view the Data in Regulation webinar on demand here
• To download the Data in Regulation report, click here

Dr. Sian Hughes Head of Data and Evidence – Equality and Human Rights Commission Sian is an economist and statistician, with extensive experience across the private, academic and public sectors.  She has a PhD in Economics, and has worked predominantly on inequality throughout her career, but also in local economic development and education.  Until 2022, she was a lead analyst at the Department of Health and Social Care, directing work on population health, health inequalities and vulnerable groups. She is currently the Head of Data and Evidence at EHRC, directing the analytical function and delivering an analytical change programme to reinforce the role of data and evidence in its regulatory model.

Naomi Nicholson Regulatory Consultant Naomi is an experienced regulatory leader, currently working as Director of her own regulatory consultancy. She has 12 years’ experience across healthcare and education regulation and has held positions in government bodies for 15 years. Naomi’s regulatory roles include directorships at Ofqual and the Health and Care Professions Council. She is also a founding trustee of the Institute for Regulation where she established and chairs the Institute’s Equality, Diversity & Inclusion Group.