Understanding scientific studies

Every day there are various newspaper, TV and online pieces about the health affects of various foods and medications. Often the claims we see today will conflict with another that we have seen before. Who should we believe?

How to read newspaper headlines

The first rule is don't believe anyone initially. Don't be fooled by the apparent credentials of the authors of the piece, or the scientific study that often underlies the headlines. A professor, doctor, TV pundit, naturopath, nutritionist or dietician can, and often are completely wrong.

The second rule is that you have to check out the evidence yourself if you want to be sure. So how can we do this?

  • Look at the scientific study that has been used in the piece you are reading/watching.
  • Look at other scientific studies that cover the same topic.

Looking at scientific studies is an art in itself. You need to:

  1. Understand what type of study you are looking at as most studies have limitations, that are not very often made clear.
  2. Be aware of how often bias appears in scientific studies. 
  3. Know where to look for more scientific studies and other sources of reliable information on the topic being discussed.
  4. Be able to interpret the numbers in scientific studies (the subject of another article you can access here).

What types of scientific study are there

At the top level you have a choice between original research papers and review papers that collate information from original research papers. The review papers are handy if you want a broad overview of a specific area of research. They are generally more readable.

If you decide to look at original research papers you need to know what you are looking at. They are generally all classified as epidemiological (research looking for reasons for various health outcomes). Within this classification you get observational and experimental studies.

Observational studies are either descriptive e.g. cross sectional studies or analytical e.g. cohort and case control studies.

Experimental studies are either animal trials or human clinical trials. Sometimes intervention trials are also used.

Those are the main types of study that you will come across. To find out more about the main features and weaknesses of each type of study, including ways to distinguish between each type look at my explanation here.

Bias in the design, publication and reporting of trial results.

Scientific studies may seem authoratitive and trustworthy, but there are at least four reasons why their findings could be suspect.

Funding.

Some studies are more reliable than others due to their funding. For instance drug company funded studies are 20 times more likely to report positive findings than independent studies(2). On top of this, the drug company studies are 30 times more likely to report positive findings in the conclusion. In other words they like to conclude with the positive points.

Study design

The reason why drug companies find so many positive findings is partly due to the way they design their studies. If you pick your subjects carefully and control their environment in certain ways, you can do a lot to skew the results in favour of positive findings for a drug. There are countless examples, but let me give you one from the world of athletics. Say I was testing different types of running shoe and their affect on performance using a treadmill. There are a lot of factors that could affect the performance that have to be accounted for. For instance heat and humidity in the room, thirst level of the subject, psychological factors such as if anyone is watching, fatigue due to repeated trials, athlete motivation if he /she believes one product is better than another. Although in theory all these factors should be controlled for, most studies can't control them all. This may be due to simple lack of time or money, but it is easy for bias to creep in when money or reputations are involved. All the experimenter has to do, is forget to control for some factors and the results can become innaccurate.

Publication bias

As well as study design there is the problem of publication bias where studies with negative conclusions don't hit the press. Recently there has been progress in getting drug companies to register their studies for publication before carrying them out, so that they cannot bury bad results so easily. Still with the power and flexibility of study design still firmly in the control of drug companies it is worth being sceptical about most drug company funded studies that come to positive conclusions about the effects of drugs.

Results vs conclusions

Most commentators, myself included, don't always have the time to check all the detail of scientific papers. Summaries and conclusions are often used without looking at details such as the trial setup, numerical results and statistical analysis.

Of course the above is not a reason to ignore the results of scientific papers, as they at least grapple with the truth of the matters in hand. This contrasts with most of our tabloid, broadsheet and online journalists who just pontificate based on prejudice rather than spend time looking at what evidence is available. Still it is important to be aware that the cloak of respectability given by quoting a scientific study is not always what it seems.

Where to look for more information

I used to get frustrated by Pub Med articles that did not have free access. All you can see is the study abstract, not allowing you to check on the details that underpin the conclusions. As you should realise, just because a study has a particular conclusion does not mean that it is right or even reflective of the results that they found. Also you cannot assess how well the study was conducted and analysed.

However, there are nowadays plenty of online resources that should provide material for most areas of interest. They include:

  1. PubMedCentral: http://www.ncbi.nlm.nih.gov/pmc/ a free access site
  2. PubMed: http://www.ncbi.nlm.nih.gov/pubmed you can access study summaries, but may have to pay for full access to the studies.
  3. Public library of science PLOS: http://www.plos.org/
  4. The Cochrane Library: http://www.thecochranelibrary.com/view/0/index.html
  5. British Medical Journal open access: http://bmjopen.bmj.com/
  6. Google Scholar: http://scholar.google.co.uk/
  7. University libraries.

References

1. http://en.wikipedia.org/wiki/Animal_testing#Species

2. http://www.drbriffa.com/2007/12/03/scientists-appear-reluctant-to-admit-...