There’s no shortage of exaggerated health claims and nutritional advice on the Internet. Many of these go unchecked or verified. Since launching Lifetime Daily, it has been our mandate to provide health and wellness information that’s authoritative, credible and importantly, backed by science. As a health and wellness publisher, it’s our responsibility to you. That’s why we were eager to speak with Dr. Tim Vines, PhD and consultant who specializes in evaluating scientific studies.
In the world of scientific papers, the peer review process is used to assess the quality, validity and originality of research before it’s published in a medical, health or psychology journal.
When it comes to assessing research quality, Dr. Vines has a razor sharp eye. Here, he share his top tips for evaluating the validity of the health claims you read online. Welcome, Dr. Tim Vines.
Q&A with Dr. Tim Vines
Q: A lot of the health and nutrition information on the Internet is closely tied to marketing firms and/or lobbyists. How can people apply a more discerning eye to what they read online? What are some of the red flags to watch for?
A: There are a few things to question:
Are you encouraged to buy something after reading the study?
If you’re reading about a research finding on a website, and it links to a place where you can buy something, that’s a big red flag. And it may be that they have a very convincing sounding study, but that study was likely done with the intent of finding a specific result, which in turn promotes their product.
Where has the study been covered?
Look for studies that are covered by major news publications, where scientific journalists are writing about the research. It makes it much more likely that it’s a real, properly conducted study. If it’s just one person, talking about it on a blog, it’s less likely to be a valid result.
Have any experts commented on the research?
Another thing to look for is comments from experts who were not directly involved in the research. It’s a good sign if a practicing and well-respected expert in the field or research community has evaluated the research and can speak to the strength of what’s presented.
Who has funded the study?
Looking into who funded a study may also give you some clues as to whether or not the study has financial or marketing interests. Unfortunately, it can get quite murky because certain groups that try to peddle a product create funding organizations or websites that look and sound legitimate. But, they’re usually very focused on a particular research outcome. If that sentiment isn’t echoed elsewhere, and they claim they’re being suppressed by the scientific community, it’s probably best to stay away. The scientific community loves and gets excited about new ideas, but the ideas need to be backed by strong evidence.
Q: What are hallmarks of a credible and trustworthy study?
A: These are some specific things to look for:
Sample Size
Sample size is the number of people involved in a study. The general rule of thumb is that if the sample size is less than 25 participants, you should take the results with a grain of salt.
When the sample size is small, you can actually end up with surprising outcomes that are the result of chance. Take flipping a coin, for example. The probability of getting heads or tails is 50/50. But you’d only get close to a 50/50 ratio after completing between 50 and 100 flips of the coin. Let’s say you completed 10 rounds of 25 coin flips, but only reported the one round where you got 23 heads and 2 tails. The result would be quite surprising for people. But, it’s a misleading result. The underlying chance of achieving heads or tails is still 50/50.
Effect Size
Effect size is the difference in magnitude between the results of two groups being studied. We often read news headlines like eating tomatoes raises the risk of skin cancer by 50%.
But 50% may just account for the small percentage difference between the two groups. For example, if one group ate fewer tomatoes but 2% got skin cancer, and the other group ate more tomatoes but 3% got skin cancer, that’s a 50% increase in skin cancer between the two groups. But overall, the effect of eating tomatoes on getting skin cancer is quite small.
Reasoning for the Finding
Take a close look at the findings. Evaluate whether the proposed mechanism in which the researchers arrived at their finding seems plausible. If the study concludes with an inability to explain why the researchers arrived at a particular finding, their exciting result may just be a fluke.
Q: Give us a quick primer on the different types of studies and their merits.
A: These are the general types of studies that you’ll come across:
Observational Studies
With observational studies researchers don’t make an attempt to change the behavior of the population studied. They simply observe the population and survey them about their health, and then try to make an inference about what causes particular conditions. These types of studies are good at identifying trends, but they can’t assign cause and effect.
Case Control Studies
Case control studies are another type of observational study where two groups of people are compared. One group has the condition of interest (i.e., type 2 diabetes), the other group does not.
But the other group has very similar characteristics in terms of age, gender, location, occupation, etc. to the first group. Researchers then try to work out what the difference is between the two groups because the systematic difference may be what’s associated with causing the condition of interest.
Cross-Sectional Study
Researchers collect data on the whole population at a single point in time and then look at the prevalence of a particular health condition. If there’s lots of lifestyle and health data available for the same group, the researchers can see which lifestyle choices and diseases are associated with the condition.
Longitudinal Studies
In longitudinal studies, researchers closely follow study participants as they go through their lives. They look at the features of participant lifestyles and then see who ended up with the condition of interest (i.e. heart disease, diabetes, stroke).
Randomized Controlled Trial
A randomized control trial is a more rigorous way of testing whether a particular behavior or feature of our life is associated with developing a condition. In this type of study, people are randomly assigned to groups. For example, one group may be asked to jog and the other group asked not to jog. Researchers then follow their outcomes and analyze who ends up contracting a condition. This is a powerful setup because the two groups should otherwise have similar characteristics.
If you find that the jogging group has lower rates of heart disease compared to the non-jogging group, then that’s evidence that jogging and lower rates of heart disease are related.
Systematic Reviews
Systematic reviews gather up all the evidence from many different studies, and decide whether the sum of all this evidence points to a particular outcome. A systematic review that’s receiving coverage in popular press is likely reporting a real phenomenon.
Q: People are often confused about food and nutrition science in particular because it’s rife with flaws and contradictions. Why is nutritional science so complicated?
A: I think there’s too many preliminary results being reported and then amplified in the popular press. When readers hear about or read something exciting, they broadcast it immediately even though there hasn’t been enough time for researchers to conduct follow-up or confirmation studies. In general, nutrition is hard to record and measure.
In addition, people’s recollections aren’t great, and what people say they’re doing when they’re part of a study compared to what they’re actually doing may not always be closely related. When large studies are conducted, a lot of different things are looked at, so you can end up with relationships between things that occur just by chance rather than there being an underlying cause and effect.
As with other areas of research, the best evidence of a food’s effect on some aspect of health will be found in a systematic review on the topic. Perhaps disappointingly, the answer will often be it’s complicated: people’s bodies and lives differ in so many ways that finding a new, universal effect of a particular food on health is just very unlikely.
Ultimately, the best source of advice on diet and nutrition comes from large professional bodies or government organizations rather than individual experts, as the former represent the consensus across many people with detailed knowledge of the relevant research. Since this consensus takes years and a lot of research to develop, these organizations may not have much to say about new dietary trends. Until the consensus emerges, it’s best to be skeptical of all claims about wonder foods or revolutionary diets.
