Making Sense of Statistics

This guide is not a lesson in statistics. It provides the questions to ask and identifies the pitfalls to avoid, to help us get behind news stories that use statistics - stories like “Diabetes drug raises death risk by 60pc”, “Gender pay gap still as high as 50%” and “Polls puts Tories up to 7% ahead”. Dismissing all statistics as just ‘lies’ does not help us get to grips with a story. By working through the points in the guide we can find out what they really mean. The four main points to look out for:

Remember, when statistics are quoted they are just the answer to the particular question that was asked. The first step to understanding the statistic is knowing what this question was and how it was asked. We can then ask where the results came from (for example, a survey, a trial, administrative data or a projection), how the samples were chosen and how the figures were analysed.
The results of studies are commonly captured in a single figure, but this figure will not represent everything that the study might have found. The common pitfalls to be aware of are: there is more than one type of average, extreme values might not be very likely and big, and small, numbers are difficult to comprehend without the context. Most of us don’t use millions and billions in our daily lives so by dividing them by the number of items they relate to we can make large numbers more meaningful.
A mathematical association, even if statistically significant, is not a certainty that one thing is causing another. Finding out what the confidence interval is for a result can give us an idea of how sure we are of the conclusions we have drawn. Confidence intervals give the scale of potential uncertainties in counting, measuring and observing data.
There are many alarming newspapers headlines about risk, its increase or decrease in relation to a particular factor. To understand the importance of any increase or decrease we need to know both the absolute and relative change and how large the risk was to begin with.

For the full guide please download the PDF from the right hand column.

For hard copies please email publications@senseaboutscience.org

Help us get this to every MP by donating at www.justgiving.com/makingsenseofstatistics

29th April 2010: Statisticians, journalists and scientists today launch Making Sense of Statistics, a guide that provides a few questions you can ask and outlines the pitfalls to look out for when weighing up claims that use statistics. Making Sense of Statistics is published by Sense About Science and Straight Statistics in collaboration with the Royal Statistical Society.

Alongside Making Sense of Statistics a short guide on Making Sense of Statistics in an Election is also released. With just a week to go until the polls the guide gives four points to keep in mind as you read the manifesto pledges, watch the final televised debate and decipher the campaign promises.

From contributors:

Leonor Sierra, Science and Policy Manager, Sense About Science: “While statistics are sometimes hyped and sensationalized, they can also test and debunk arguments. Knowing the questions to ask and the pitfalls to avoid helps us to work out whether the figures in the headlines matter to our lives and to society, from health screening to the economy.”

Nigel Hawkes, Director, Straight Statistics: “Statistics help to make sense of a confusing world. But the sheer number and variety of statistics also provide great opportunities for misrepresentation or selective quotation. Getting canny about these tricks should be part of everybody’s armoury.”

David Spiegelhalter,Winton Professor of Public Understanding of Risk, University of Cambridge: “We shouldn’t just accept the numbers we’re told as absolute truths, but ask where the evidence comes from, what it relates to, and even what we’re not being told. Numbers can’t tell us everything, but by understanding a bit about statistics, uncertainty and probability we can look critically at stories in the news.”

Michael Blastland, writer and broadcaster: “Taking numbers for granted is naive - you become a sucker for spin. But treating them all as so many lies, turning away in cynicism, is to give up on every political, economic or social argument you follow, every public cause you love or hate. The middle way is the only way: to learn how numbers work.”

Andrew Garratt, Press and Public Affairs Officer, Royal Statistical Society: “A great many of the decisions we make - or are made for us by government - are underpinned by statistics. Knowing more about how they work helps us to make better decisions and hold government accountable to the public.”

Simon Briscoe, Statistics Editor, Financial Times: “The advent of widely available, good quality data is a new phenomenon. Its impact on the way we live will be just as great as the internet or mobile communications. If you don’t ‘get it’ and know how to use it, your quality of life and understanding of the world around you will suffer. And it is fun to have the insights into our world that numbers offer.”

Christina Pagel, senior research fellow, Clinical Operational Research Unit, UCL: “I think that decisions on how to use modern scientific advances, whether in health, cutting edge technology or the environment are so important that everyone in society should have a say. To do this, we all need to be informed and to understand how to interpret the scientific evidence presented so that we’re not unfairly influenced by those presenting the data.”

Shaun Treweek, senior lecturer at the School of Medicine, University of Dundee: “Numbers and statistics can help people to make informed decisions about what tablets to take, what food to eat, or what car to buy but they need to be handled with care because they can also be misleading. This booklet should make writing about, or reading about, statistics and numbers a little bit easier. That has to be a good thing.”

Last updated: June 23 2010