Medical fact or media fiction? How to tell if research is reliable

Most of the health stories you read in newspapers or magazines are built around some sort of medical research. But how do you know if the research itself is any good or if it has been accurately reported? Jim Pollard explains how to read between the lines.

pills on trial Firstly, what sort of research is it? It's amazing how many medical breakthroughs have been announced on the back of experiments on mice, rats or on body tissue alone. If it is that sort of research, you can bet it won't be available for human treatment for years, if ever.

The most important sort of research so far as the treatment or prevention of disease in humans is a clinical trial — a trial involving real people and real treatments. But some trials are more reliable than others. When you see the headline 'Irradiated cow dung can change your life', here are the questions to ask before you rush off to your local farm.

1. Is the trial randomised? This means patients are selected at random not according to the doctor's bias which can lead to those who are most likely to be 'successful' patients being selected.

2. Is it 'double blind'? Double-blind means that neither the patient nor the doctor or other professional observing them knows whether that particular patient is receiving the treatment on trial or not. Again this reduces bias.

3. How big is the study? Small studies are unreliable. The rarer the problem under investigation, the more unreliable a smaller study is.

4. Are the results statistically significant? You can do a degree in stats and still be left with questions about statistical significance but put simply, findings are statistically significant if they cannot be attributed to chance alone. Here's an example.

The odds on getting a head or a tail when tossing a coin are evens (50-50). That means that if you toss a coin 100 times, you'd expect 50 heads and 50 tails. But in reality this is probably not what would happen, it would be 51-49 or 52-48, perhaps more. However, that would not mean that the odds on the next toss would be different, they are always 50-50. Statistical significance tries to allow for this variation between theory and reality in an experiment.

Example: 100 men and 100 women are given an IQ test. The men score 102, the women 104. Does this prove that women are better at IQ tests than men? Probably not. It's not statistically significant. However, if the men score 98 and the women 120 you might be tempted to draw different conclusions.

The allowable variation depends on what is being studied and how but 5% is common. Good research will make it clear that its findings are statistically significant.

5. How was study conducted? The way an experiment is done will nearly always distort the results in one way or the other. You need to look at how an experiment was done and see where the problems and distortions might arise.

Patient questionnaires, for example, are unreliable — one man's 'excruciating' pain is another's 'mild niggle'. In an example like this it might also depend on how the questions were asked — if by a female researcher, men might play down the pain.

Even asking someone if they feel better is fraught with problems. People often feel better simply because someone is bothering to ask them.

6. Did the trial really measure what it claims to have measured? (Or at least, what journalists claim it measured.) Take the example of the IQ test above. If men scored 98 and women 120, the headline could well be 'women more intelligent than men' but that would not be true. It presupposes that IQ tests measure intelligence which is debateable. All that can accurately be said is: 'women better at IQ tests than men' — trouble is that's not such a good headline

7. Who was paying for it? If research findings appear to be in the commercial interest of the firm who sponsored them, it doesn't mean they're not true, it simply means you're right to be sceptical about them.

The increasing commercialisation of science with research departments dependent on the publications of their staff for funding can also encourage the 'spinning' of results.

8. What's the real risk? In the reporting of stories, journalists will often talk about 'risk'. The question is: is this a relative or absolute risk. Absolute risk is what you should be most interested in but relative risk, because it's more dramatic, is what is usually reported.

Example: you have a 1 in ten million chance of catching X. Research shows that drinking tea doubles your chance of catching X. This creates a great headline: 'tea-drinkers at twice the risk of X'. But what does it really mean? Doubling the risk means the chance is two in ten million or one in five million. In other words although the relative risk has doubled, the absolute risk is still very, very small.

If, after you've asked these eight questions, the research still looks pretty good then you can think about believing it. If you're still not sure, there's one very good place to check — here on malehealth.

Jim Pollard is editor of malehealth