Imagine you read a brief news report in a respected peer-review journal such as Nature. The report cites a conclusive study which states that by giving up eating blueberries, you can cut your risk of getting a certain type of normally fatal cancer by 90%. You like blueberries, and eat them fairly regularly.

Would you give up blueberries based on this?

Killer Blueberries

There are two obvious answers, and, at the risk of just plain speculating, I would suggest each has its typical justification.

A lot of people are going to say “Yes, I’ll give them up: it is not a big sacrifice to make, and if it cuts down the risk of a cancer by 90%, that has got to be good news.”

The No group is going to contain at least some people who are fed up with being told what’s good for them by “the Food Police”, and say “To hell with it, life is too short, I like blueberries, and am going to carry on.” (We saw a lot of this pushback recently in the UK, when the World Health Organization re-classified processed meats as carcinogens, in the same category as smoking, alcohol and asbestos. Although bacon sales went down, there were some people who stubbornly announced they were going to have a damn good fry-up Sunday morning.)

I would submit that both answers, and both justifications, are wrong, because the true response should be:

Not enough information given to answer the question.

And the reason is, you haven’t been given two very important pieces of the puzzle: the base rate and the outside view.

Base Rates

What is the base rate, and why is it important? The most significant base in this case is the prevalence of the cancer in question. Until you know that, the impressive-sounding “90%” stat is meaningless, indeed misleading.

Cancer is a big killer, accounting for nearly 30% of all UK deaths in 2011 (source:  Cancer Research UK). So it is a pretty good thing to avoid.

However, there are many different kinds of cancer, and some of them are very rare. Our imaginary news report doesn’t state how rare the no-blueberry-curable cancer is (we’ll call it NBCC), so we are free to try a little thought experiment. First, let’s speculate in the extreme….

Let’s imagine there are only 10 cases of NBCC diagnosed each year, world-wide, and the diagnosis rate is 100%. (This is a thought experiment, after all!) Given there are about 7.5 billion people in the world, very, very roughly, an individual’s odds of being diagnosed with NBCC are one in 750 million. That’s a pretty low risk factor. According to www.wolframalpha.com, the odds of you being struck by lightning in a given year are 1 in 750,000, so your risk of developing NBCC is very roughly 1,000 times less. (Aside: It is amazing how nicely numbers can line up if the author treats himself liberally to the qualifier “very roughly”.)

Certainly there are some who would still reply that death is a pretty bad thing, so every little bit of reduced risk helps. I’ll deal with this objection in a minute.

Let’s move on to the “No” group. The above thought experiment in the first instance seems to support their case, until you imagine a different base rate, swinging to the opposite extreme. Let’s assume that NBCC is one of the major cancer risks, accounting for 10% of all deaths. If you can reduce that rate by 90%, you move from a 1 in 10 chance of developing NBCC to 1 in 100. That seems like a pretty good reduction for a pretty small sacrifice, and the No’s might be well-advised to save their Food Police objections for something less lethal.

Everyone should be prepared to give up something, if the risk is high enough.

The bottom line here is, until you know the base rate, the “reduce risk by 90%” stat has no meaningful role to play in your decision making. The big number, 90%, sounds significant. In fact, it might be miniscule. We are back to “Not enough information given to answer the question.”

The Outside View

But we can go even further, by taking what Daniel Kahneman calls “the Outside View”. Basically, we need to look beyond the concerns raised by the question. Because of the way it is pitched, we focus on the negative effects of blueberry consumption: how giving them up can prevent NBCC. But what if eating blueberries are, in other ways, beneficial? Let’s flip that ’round, and pose the following question…

What are the harmful effects of giving up blueberries?

Blueberries are a rich source of anti-oxidants. Although the most recent studies are starting to cast some doubt on anti-oxidants as a miracle cure, let’s again proceed with a thought experiment. We’ll imagine (ignoring entirely their positive or negative role in cancer), that blueberries are very good for you in other ways.

In fact there is some evidence (a decent summary appears here on the UK’s NHS web site) that blueberries can reduce heart attack risk, and lower blood pressure. The same page states there is less support for some of the other hyperbolic “super-food” claims made for blueberries, but heart disease and blood pressure are good enough to make the point.

In fact…

“Coronary heart disease (CHD) is the leading cause of death both in the UK and worldwide.”

That is the first sentence on the NHS web site’s Coronary Heard Disease page. So anything that reduces CHD risk has got to be a good thing, doesn’t it?

So bring on the blueberries….

A Good Blueberry

But wait a minute, what am I saying?!

We still haven’t got enough information to answer the question. And we need to go back to base rates. How much do blueberries actually reduce risk? In what types of coronary disease? In what age groups?

We’ve just once again been mislead by the siren song of yet another pithy statistic.

So What AM I Saying?

Good question.

I am certainly not joining the ranks of those who say science can never make up its mind. Quite the opposite. There is useful information out there, and we should be guided by it, at all levels of our policy-making, be it blueberry consumption or nuclear generator construction.

We do have to take decisions, and frequently on limited information. And certainly, we cannot spend all afternoon researching the blueberry issue.

But the above exercise does serve to remind us that statistics taken out of context — stats that lack a base rate and don’t express an outside view — are both misleading and tempting. Being aware of whether the base rate is expressed, and the outside view is engaged, can help us to decide whether to trust a stat, or quarantine it until we can dig deeper, if the issue is important.

So next time you get some pithy, scientific-sounding advice, take it with a grain of salt.

Wait a minute…. ….salt? That’s bad for you, isn’t it?

©2016 Brad Varey