There is a large amount of nutrition and health information out there – and it seems that for every piece of advice available, there are at least two more that contradict it.
Even people who appear to know what they are talking about often take their opinions from other people, which can lead to all sorts of hype and belief that simply does not equate to reality.
The problem is that health and nutrition are highly complex fields and it simply isn’t possible to find advice that will work for every person in every situation.
So, what do you do?
Perhaps the most valuable approach is to look at the research itself.
There is a huge amount of research out there on health and nutrition, and more research comes out frequently. If you can understand what the research has to say, then you are in a much better position to understand the health claims that people make and whether those health claims actually make sense.
Scientific research is particularly important because it tends to be peer-reviewed – which means that the authors have to go through a review process before their work gets published.
This helps to ensure a certain level of quality.
It also means that a scientific paper will tend to be more reliable than a random article that you stumble across online.
Why Look at Scientific Papers?
There are probably two situations where you might end up looking at scientific papers when considering your own health.
The first is if you come across specific health claims – particularly from sites that actually link to scientific studies.
Companies make these claims all of the time and blogs often do too. Yet, the claims themselves are often misleading and inaccurate, even when the company references scientific articles.
One example of this is a product called Imortalium, produced by the MLM company Youngevity.
There is a huge amount of hype surrounding this product, but if you look into the science behind it, you start to realize that there really isn’t that much supporting the claims.
To start off with, the product is based around the concept of protecting telomeres.
Now that much is actual science. Telomeres deteriorate as people age and they are at the end of chromosomes. Personally, I wouldn’t call them coverings, but that’s not too relevant. Regardless, scientists do know that telomeres play some role in aging, although how much of a role is less understood.
This means that protecting telomeres may help with some aspects of cellular aging, but no one really knows how much. Protecting telomeres may also do nothing at all for aging.
The site for this particular product only lists four scientific studies as evidence that their claims are legitimate. Now, if you were considering buying this product, you might choose to look at those papers and see what they have to say.
I’m not going to go into those papers in detail here, but they don’t really support the claims of the company.
The other way you might come across scientific papers is if you actually looked for them specifically.
Personally, if I’m interested in any health issue or supplement, one of the first places I look is the research. There are a few different ways of doing this, but a particularly easy one is an element of Google called Google Scholar.
This aspect of Google searches across scientific papers and it can be quite useful. It’s also very easy to pick up because it works so much like Google itself does.
For example, a search on ‘Vitamin D Deficiency’ turns up this as a result:
Sometimes you will have access to the full paper in the outcomes, and other times only the very beginning.
In this post, I want to teach you how to read scientific papers and what the different pieces of information mean. Ultimately, taking the time to learn this is critical if you want to take control of your own health and not get mislead by false claims.
Journals and Authority
Most of the time, research is published in a scientific journal. These journals normally use a peer-review process to make sure the paper is accurate and relevant.
There are a huge number of journals out there, and there is a lot of debate about which ones are the best.
One approach (albeit somewhat controversial) is to look at the impact factor of a journal. This is an indication of how many times articles from that journal are cited.
Generally speaking, a high impact factor is an indication of a higher quality journal – although not everyone agrees.
If you are reading any scientific research, it’s worth at least considering the journal the article is published in. If the journal is obscure, the research may be low quality. However, this is not the be-all-end-all because you can get high-quality papers in poor quality journals and vice versa.
There are also some journals out there that aren’t peer-reviewed and sometimes specific article in a journal aren’t peer-reviewed.
You need to be very careful with any research that isn’t peer-reviewed because it can be misleading and even correct.
One of the best ways to find out whether a journal is peer-reviewed is to look on it’s about page. For example, The Journal of Cell Biology makes it clear that the publication is peer-reviewed.
For-Profit versus Open Access
Typically, scientific journals operate on a for-profit basis. This can make it expensive for people to get articles published, and it also means that access to the journals is limited.
Most of the time accessing papers from a for-profit journal is difficult unless you happen to have access to some journals through a university or somewhere else that subscribes to journals.
You can also pay for access to individual articles, but this ranges from around $20 to more than $100. That’s a pretty expensive approach if you want to look at multiple articles.
However, in recent years, there has been a growing number of open-access journals. Many of these still focus on having a rigorous peer-review process but make it much easier to access the articles.
If you are looking for information on any topic, open access journals are often a good place to start, because you know you are going to be able to read the articles.
Two of these journals that I particularly like are PLoS (which is actually a set of journals) and Frontiers.
Nevertheless, it is still worth seeing whether you can find any articles from high-quality for-profit journals because these do tend to have better research and higher quality information. A search in Google Scholar is a good way to do this, as you sometimes find links to the full-text article, like this:
Important Things to Consider
Science is a rapidly changing field and this means that research can get out of date relatively quickly.
A good general rule is not to pay much attention to papers that are more than ten or so years old unless there is something highly significant about that paper. Most of the time, there will be more recent papers that contain more up-to-date and relevant information.
In fact, you will often find that a research paper is completely refuted by a paper that comes just a few years later.
This means that you really do need to consider the age of a paper.
Another thing to look at is the type of paper and the processes used.
I’m going to go into these aspects in detail later on in the article, but it is important to realize that there can be huge differences in how accurate the information in an article is and what that information means.
Elements of a Scientific Paper
There are a lot of variations between scientific papers, particularly when you are looking at papers from different journals.
Yet, despite this, there are also many elements that are similar across studies. In fact, their consistent format means that it is easy to find the specific pieces of information that you are looking for.
Abstract
The abstract is normally the first part of a scientific paper and it is all that you can see if you do not have access to the full paper.
The concept of an abstract is that it provides basic information about everything in the paper, including the rationale, methods and results.
Sometimes this will be written as a single paragraph, like this:
But other times it may be broken down into sections, like the abstract below.
Differences in format and length of the abstract are normally the result of the journal that the article is published in, as each journal has their own requirements.
In essence, the abstract acts as a brief summary of a research paper, including the approaches that were used and the general outcome.
However, abstracts are also limited.
Because of how short they are, abstracts often miss key things about the paper, including information about how reliable the observed outcome or outcomes were.
Additionally, the authors will sometimes write the abstract to highlight specific results while minimizing the importance of others.
So, always read the full paper if you have access to it.
Background/Introduction
The background section of the paper looks at the history of research in the field, and why the question that the paper addresses is important.
Sometimes this section is very short and rarely it isn’t present at all.
However, normally this part of the paper is comprehensive because it provides valuable information.
The background section is important for showing the trends in the field and where the scientific study fits in with current research.
In this section, the authors will also reference other studies, giving readers the chance to look up some of these studies if they choose to.
For example, this is the majority of the background section from a paper in Circulation on Vitamin D and Cardiovascular Disease.
The very first part of this background section offers a broad overview of the field while the rest of the section considers much more specific components.
You can see that in this case the word background is never used, yet the section does still act as an important background to the topic.
Often the background section is used as an illustration of why a particular topic is important, which can act as justification for the research.
It is worth noting that the background section does not necessarily cover everything in the field, and normally authors will choose only to discuss what is relevant to their study.
Additionally, authors will sometimes pick parts of the field that support their study idea and ignore parts that do not.
The peer review process does limit this from happening somewhat, but it is still worth being aware of.
Methods
The methods section can also be the most confusing section of a research paper, particularly if you aren’t familiar with the field or the research approaches being used.
When you’re reading a research paper, it’s tempting to skip past the methods section entirely, but I really don’t recommend it.
The methods are important because they show how a study was done.
That information is what tells you whether the outcomes of the study make sense or not.
For example, when there was a lot of hype about green coffee bean extract and weight loss, one particular study was frequently referenced and talked about.
However, most people who referenced the study never bothered to pay attention to its design, which is a pity, because the study was horribly designed.
This is a section of the methods for the study:
The first part sounds fine, “randomized, double-blind”, and this is actually a very desirable type of study. However, the rest wasn’t nearly so good.
To start off with, there were only 16 participants (6+4+6), which is a very low number for any experimental study.
The study involved three types of treatment:
- High dose
- Low dose
- Placebo
Normally, a study would randomly assign people to each of those three groups (without telling the people) and then watch the results. In a double-blind study, neither the participants nor the researchers would know who was in which group.
So, with those treatments, one group would get a high dose of green coffee bean extract, one would get a low dose and one would get a placebo, which would contain none of the green coffee bean extract. Ideally, all of the treatments would look the same and any pills would be the same size. In that case, the placebo group is a control, helping to prove that any observed effects were because of the green coffee bean extract, not the research design.
That’s a solid scientific design, but it’s not what this study did.
Instead, all participants were given all of the treatments, just in different orders.
The end result was that the participants lost weight. So, the authors argued that the outcome showed that green coffee bean extract actually works.
Here’s the problem with that.
First, there is no indication of which treatment (high dose, low dose or placebo) caused that impact.
Second, it’s very likely that the weight loss had nothing to do with the green coffee bean extract at all.
Instead, it is most likely connected to the placebo effect.
Basically, the placebo effect refers to how effects of a treatment can be seen even when people were simply given a placebo (normally a sugar pill or a saline injection) instead of actual medication. The placebo doesn’t cause any physical change directly, but instead, the observed outcomes are connected to the beliefs of the patients.
The placebo effect is actually very powerful.
Any half-decent experiment will include a control group who receives a placebo to take this effect into account.
The way that this particular study is designed means that there is no control group at all.
So, all the study shows is that participants lost weight. That could be because of the Green Coffee Extract or it could simply be because the participants were in a study that weighed them regularly.
Think about it… even if you were told not to change any part of your diet or eating habits, if you were in a study that involved weighing you, you would probably be more careful in what you ate. Just being weighed alone would make you more aware and more self-conscious and that would result in weight loss.
In this case, the design of the study was so bad that it was actually retracted, about two and a half years after it was published.
The issues from the study were always evident from the methods and the study itself is a real lesson on the importance of reading the methods carefully before believing what any study has to say.
Results
The results section of a paper focuses on the actual numbers and outcomes that the study found.
This can often end up quite complex in the cases of studies that examined a lot of information and most of the time those numbers won’t mean all that much to you as you are reading the study.
For example, it’s pretty common to see tables like this one in the results section:
I’m not going to go into how to read these types of tables too much, but I do want to point out the right column, the one that says P.
In statistics, that P refers to probability. Specifically, it refers to the probability of something occurring by chance. So, a value of 0.53 indicates that there is a 53% chance of the observed outcome occurring randomly.
By convention, if the P value is less than 0.05 (i.e. 5%), then it is considered to be statistically significant.
That means that a P value of 0.05 or less means that a particular difference is likely more than just chance.
The table from this particular study was looking at the level of vitamin D in the serum of participants. Participants were broken down into two groups, one with vitamin D levels below 15ng/ml and ones with vitamin D levels above or equal to 15ng/ml.
The first line (highlighted in red) looks at the age of participants. The P value indicates that there was no statistical difference in the average age between either group. You can actually see that in the fact that the value for each serum level was the same.
In contrast, the third line (highlighted in green) looks at body mass index. Here, the people with the higher levels of vitamin D tended to have a lower body mass index, while those with less vitamin D in their serum tended to have a higher body mass index.
In this case, the P value indicates that the difference is statistically significant.
So, this would suggest that there might be a relationship between body mass index and vitamin D concentrations.
Just from those numbers, it isn’t clear what the relationship is, but this is where the context of the paper and what the authors discuss comes into play.
Generally speaking, you won’t see a discussion of what a given set of numbers means within the results section. This type of discussion is normally reserved for the next part of the paper, which we will come to in a minute.
The results section can be a little unwieldy at times, and it can be challenging to understand, particularly if the tests were complex.
However, the results do give a clearer idea of the context of the outcomes that the authors found and this can be important for working out what the authors actually found as the result of their study.
Discussion
The discussion section is where the authors talk about the results of their study.
This includes talking about how the different individual results relate to one another, what their implications are and how they relate to the topic as a whole.
This tends to be one of the longest sections of a scientific paper and it is also very important.
For example, in the particular paper we have been considering, the authors use the discussion section to compare different possible mechanisms that might link vitamin D deficiency with cardiovascular disease.
Often the authors will also use this section to talk about how their work relates to other work that has been done in the field and to make recommendations for future work.
Additionally, the discussion section of a paper often includes a limitations subsection (although this may sometimes be its own section).
This subsection or section considers the weaknesses of the study, including limitations in how widely its results can be applied.
Having limitations isn’t necessarily a bad thing and almost all scientific studies will have some limitations (although the authors do not always discuss them). The process of discussing limitations can help to guide future research and to indicate how the paper should be interpreted.
Generally speaking, a good quality paper will have a limitations section.
The presence of that section means that the authors have actually thought about the strengths and weaknesses of their paper and acknowledge them. That’s much better than a paper that pretends it doesn’t have any weaknesses.
Conclusion
The final section is the conclusion, which is basically a summation of the paper.
Often the conclusion section is short, and it will frequently focus on the messages that the authors want people to take away from their paper.
If you’re short on time, the conclusion section is a good piece to read, but like the abstract, it is missing many important elements of the paper.
Types of Studies
The sections I’ve talked about here are pretty common in research papers but they aren’t always present and papers often have other sections and subsections as well.
The specific journal and the decisions of the authors can influence this, but so does the type of research.
There are actually many different types of scientific papers out there, and they tend to have different strengths and weaknesses. I’m going to touch on some of the main ones here.
Primary Research
The term primary research refers to the collection and analysis of data by the researcher.
One of the most common approaches to this is where the research conducts an experiment. This might involve something like looking at whether taking a vitamin D supplement decreases the risk of a heart attack.
In this case, the researcher might have multiple groups taking different levels of vitamin D along with one group taking a placebo. The researcher would also need to have some way of measuring or estimating the risk of heart attack.
Experimental studies tend to be more resource intensive than observational studies, which is one of the reasons why they are less common.
Additionally, experimental studies tend to require more ethics approval and often end up using small groups of participants because of resource limitations.
I’ll discuss the implications of a small group of participants later on, but this certainly is a problem with experimental studies.
Experimental studies are the most effective way of testing cause and effect because they allow the researcher to break the participants into different groups and put the groups under different treatments.
For example, one group might get drug A, one might get drug B and one might get a placebo.
At the end of the study, statistical analysis of the outcomes will show what actual impacts the different drugs had.
One of the most effective form of experimental study is known as a double-blind randomized placebo-controlled study (or some combination of these terms) (1).
Double-blind means that neither the researchers nor the participants know what groups the participants are in.
This approach helps to make sure that all groups receive the identical treatment.
Randomized means that people are randomly assigned to treatment groups, which reduces the potential for any differences between groups.
Finally, placebo-controlled means that there is a control group that is receiving a placebo treatment.
When this type of study is designed well it provides the strongest possible indication that there is causation, rather than just correlation (2).
While experimental studies are desirable, they are not possible for all topics.
For example, some types of experimental studies would be unethical to conduct, such as exposing one group to abuse and not another group.
An alternative approach to primary research is where the researcher does not manipulate any variables but instead measures different outcomes. This approach can be referred to as observational research.
This form of study is very common because it is relatively inexpensive to conduct and can use data gathered for other studies or other reasons.
One of the reasons that observational research is so popular and so powerful is that it can include large groups of people, many more than most other forms of research.
This allows for the detection of patterns and trends that could not be found otherwise.
However, observational research has a major drawback.
The issue is that it can’t detect causation.
One of the key examples of this is the link between heart disease and saturated fat.
This link has largely been created through observational studies.
Yet, saturated fat has been linked to heart disease for so long that most healthy people (excluding people on Paleo or high-fat type diets) avoid saturated fat in their diet.
So, these observational studies have actually found a link between unhealthy lifestyles and heart disease, rather than specifically between saturated fat and heart disease.
Some observational studies get around this problem through statistical analysis by controlling for certain variables.
For example, this could involve testing for a relationship between saturated fat and heart disease while controlling for factors that could indicate a poor lifestyle, such as lack of exercise or BMI.
This can only be done if the data actually exists, but it would provide a more accurate picture of whether saturated fat itself plays a role in heart disease.
Secondary Research
In secondary research, the researcher is analyzing data that he or she did not gather.
For example, large data sets (like Census data) offer fantastic chances to analyze data and look for trends and patterns. However, that data tends to be observational in nature and suffers from the same limitations.
Another type of secondary research is a meta-analysis.
In this case, the authors act to collate and examine the outcomes of studies within an area and this may include testing for significance.
This can be a particularly important way to find out the current status of research, particularly when there have been a number of contradictory studies in a given field.
For example, a meta-analysis might look at all the research done on the link between coffee and weight loss, and report how many studies found a positive effect and how many didn’t. It might also look at the size of the observed effects and how many of those studies were of high quality.
Additionally, a meta-analysis will often provide information about the quality of the studies in the field and an indication of what direction research should move in.
Even though they do not do any experimental research, meta-analysis papers can be very valuable as they allow for a wider understanding of a field, which can be critical for fields that are new or complex.
However, a meta-analysis does go out of date fast because there are frequently new studies published.
Another type of research in this field is a literature review.
This type of study is similar to a meta-analysis in many ways.
The biggest difference is that a review paper doesn’t attempt to quantify the progress in the field.
Instead, it acts a discussion of the field, including current and future directions.
Often this type of paper will critique some of the research in the field, pointing out its errors, and discussing what is still unknown within the field.
This type of paper can be particularly relevant for understanding a given field.
Finally, some scientific papers consider the theory around a particular area.
There is a wide variety in these types of papers, but they will often involve examining current research in a given field, and proposing a mechanism of action.
For example, a theory paper might look at how green coffee extract may help people to lose weight.
While the paper does not add any evidence to the debate, the theory that it discusses may be important for other research in the area.
Things to be Aware of
Sample Sizes
Studies vary in the number of participants they use due to a range of factors.
Often, studies will have a low sample size because of funding issues or because it is an early study into a field.
Studies that have low sample sizes are still valid, and they can provide important information, particularly when there are no better studies available.
However, care has to be taken when reading and interpreting studies with small sample sizes, because their outcomes may not be reliable.
This is because there is a large amount of variation between people and the fewer people who are part of the study, the less variation that will be captured.
This means that the outcomes of small studies will not necessarily extend to the full population.
For example, if a supplement helps ten people to lose weight, there is no guarantee that it will be effective for the majority of the population.
Sample Selection
The way that the sample is selected and what the sample is has a lot of implications for how reliable a study is.
Ideally, the sample should be selected randomly, particularly for experimental studies, although this will not always actually be the case.
For example, many people conduct scientific research as part of an academic role, like being a professor at a college.
Because of this, researchers will often use students at colleges as the participants for scientific studies, as they are readily available and can often be offered extra credit in exchange for participation.
This will not necessarily produce invalid results, but it is important to take into account when interpreting the outcomes of a study.
One of the reasons for this is that there are significant differences between college students and the rest of the population, such as college students tending to be younger and often having unpredictable lifestyles.
Keeping the sample selection in mind is important when reading any scientific study because the outcomes from one population group do not necessarily apply to a different one.
Additionally, another issue arises when people opt into a study.
This is particularly significant for survey-based studies, such as when people respond about their political convictions.
The problem is that when people have to opt in for a study, you end up with people who care about the issue at hand and not many that don’t.
For example, an opt-in study that was examining ways to improve health would end up with participants actively interested in their health.
After all, people who weren’t interested in improving health probably wouldn’t be interested in the study either.
If you conducted a weight loss study on this group, you might find that they were able to follow a diet well and lose weight effectively.
However, this outcome may not apply to parts of the population who do not care as much about their health.
Control Groups
Any experimental study should have a control group and this should be relevant to the study and what the authors are trying to determine.
A control group gives the study authors a baseline to compare against, and can help to eliminate many potential sources of confusion.
This is particularly important because the process of an experiment can often produce an effect.
For example, if a group of people is told they are being given anti-depressives during an experimental study, there will be a significant decrease in depression among the sample group even if they are only taking a placebo (i.e. a pill that does nothing – often a sugar pill).
This is known as the placebo effect.
Likewise, some studies on dieting find that even the control group loses weight because they are more aware of their habits as a result of the study and end up changing their behavior unintentionally.
This means that any study without a control group is very suspect, and this should be taken into account when reading their conclusions.
For that matter, some authors (including Mark’s Daily Apple) suggest that diets may even work primarily because of the placebo effect.
Animal Studies
Research studies on animals are sometimes used instead of studies on humans.
There is a range of reasons for this, which include the fact that it is easier to get ethical consent for animals, animals may be less resource intensive or as a way to indicate safety before human studies can begin.
Animal models provide important information about how the human body is likely to respond to a given treatment, but they are not definitive.
There are differences between animal models and humans, and this means that studies may find an effect in animals that is not present in humans.
This is something to be aware of when reading scientific studies, and in general evidence from animal studies is far from definitive.
Confounding Factors
Scientific papers are one of the most reliable sources of information out there, but as you can see – there is still a lot of potential for incorrect and even misleading information.
The more time you spend reading and trying to understand scientific papers, the more chance you will have of picking up the evidence that they present and any issues that are in the paper.
I’ve talked about a number of different ways that a scientific paper might be confusing or misleading, and before I close, I want to introduce one final term: confounding factor (or confounding factors).
Research studies often involve the consideration of the relationship between two factors, like age and height.
A confounding factor is an additional factor that can influence the outcome.
Let me illustrate.
A classic example of this is the relationship between height and test score at a high school.
Logic tells us that height isn’t going to affect test scores – because there isn’t a mechanism where it could.
Yet, if you were to do an experiment where all students in a school were given the exact same test, you might end up with a plot that looks like this:
The graph is a bit simplistic – but you get the idea.
There isn’t actually a causal relationship between height and test score, but it looks like there is because of a confounding factor.
In this case, the confounding factor is age. In general, as people age their height increases, especially when they are young.
Additionally, older students would tend to perform better than younger children at the same test when all else was equal.
You can find examples of this pattern at the site Tyler Vigen, which focuses on correlations that are simply statistical probability – nothing more. One such correlation is below:
Now, it is possible that these two areas could slightly influence one another. But, there’s no way one is causing the other. Instead, the outcomes are either the result of chance or some external factor (probably the economy).
Because of this, scientific research has to take confounding factors into account.
This can be done through some statistical approaches.
The key phrase to look for in any scientific study is ‘controlling for’ or ‘controlled for’ (or a similar variation).
For example, in this case, I would test the relationship between test score and height when controlling for age. This would show whether there was any relationship between the two when the influence of age is taken out of the equation.
When reading a paper, it’s worth taking the time to consider whether the authors actually took all of the factors into account or not. If they don’t mention a potentially confounding factor, then there is good chance that it wasn’t considered in their analysis, meaning that all of their conclusions could be wrong.
It’s also worth noting that confounding factors are most common in observational studies – where none of the variables were manipulated.
In contrast, the design of an experimental study tends to hold many confounding factors constant, which makes it easier to account for potential issues. The use of a control group also plays a major role in reducing confounding factors.
But, in an observational study, it can often be difficult to identify all of the potential confounding factors, particularly if the link between the factors and the outcomes isn’t immediately obvious.
One example of this is the relationship between red meat and health issues like obesity and heart disease.
Observational studies do often indicate that people who eat red meat tend to be more at risk for a range of health issues.
However, one issue with this is lifestyle.
For quite some time, people have been consistently told that red meat tends to be bad for your health and if you are going to eat it, have it rarely.
This means that in general, people who are living healthy lifestyles and eating healthy food will tend to avoid red meat, because they have been told it is associated with bad health.
In contrast, people who have less focus on their health will tend to eat more red meat.
This means that when you look at the data, there is a relationship between red meat consumption and health, but one doesn’t cause the other.
Instead, people who are healthier in general tend to eat less red meat, while people who are less healthy tend to eat more (this is a huge generalization of course).
As a result, factors such as lifestyle are huge confounding factors in the study of whether red meat actually does have negative health impacts.
A Primer on Significance
In general, science is focused on understanding the world around us and research is a key factor in achieving this goal
But, the world is complicated.
Scientific research attempts to sift through many complex and interacting factors to work out what is actually going on.
Because of this, you will find that even the most well-designed scientific studies may contradict one another.
In a similar way, researchers cannot ever be entirely certain that the effects they see in their research are actually real.
For example, many studies that compare two (or more) groups will find some difference between them, but this difference may be the result of chance and nothing more.
Because of this, statistical tests are used to determine whether the outcome was likely to be the result of chance or something else.
There is a range of tests that are conducted, and I’m not going to go into them here.
However, the key piece of information that scientists look for is whether the outcome of the study is statistically significant.
Think of it this way:
If you took two people and put them on different diets, then measured weight loss at the end of the period – would any difference tell you anything about the diets they were on?
Probably not.
After all, there are many differences between the individuals and these could have influenced, or caused, the weight loss.
However, if you did the same experiment with 200 people and one group consistently had more weight loss than the other, then it is likely that the difference is more than chance.
A statistical test will look at the weight loss experienced, the group size and other factors to work out what the probability of the outcome being caused by the experiment versus chance.
By convention, most scientific papers use p = 0.05.
This means that there is a 5% probability of the effect being nothing more than chance.
For an effect to be significant, it must be at least 0.05, and preferably under.
So, for example, you might see a research study that compares weight loss between two different diets, like a high-fat diet and a high-carb diet.
The authors might say that there was a statistically significant difference between the two groups or just a significant difference between the two groups.
That indicates that the p-value was less than 0.05 (or whatever the study had set).
That outcome would show that one diet really is better than the other.
However, it is important to remember that there is always the chance that the authors are wrong.
If the p-value is 0.01, then there is still a 1% chance that the difference comes from chance. That’s why studies focus on using large groups wherever possible, as this helps to decrease that risk.
Basically, the lower this number, the more reliable the outcome of the test is.
This is statistical significance and studies may say that they observed a statistically significant trend.
Sometimes, a study will say that they observed a trend, but it was not statistically significant.
Be very careful about any trends that aren’t statistically significant, because this basically says that the authors observed a trend, but it may not mean anything and it may not even be real.
Graphs and Significance
You can get a sense of significance and how good the measurements in the study were from the graphs that a study presents.
This also applies to any graphs that you see online.
For example, you might see a graph like this:
This shows that people on the high-fat diet lost more weight than the ones on the high-carb diet.
But, on its own, the information is actually meaningless.
The graph is basically showing averages. So, if there were 100 people in each group, the average weight loss for the high-fat diet was 22 lbs, while the average loss for the high-carb diet was 17 lbs.
To know whether this means anything, you need an idea of how much variation there was.
For example, if the high-fat diet had losses ranging from zero pounds all the way to 50, and the high-carb diet had a similar range, then the difference in averages probably doesn’t mean all that much.
One tool used to look at this on graphs is error bars.
There are actually a few different pieces of information on graphs that can be used to form error bars, but one of the most common is the standard error. I won’t go into the calculation for this, but in general, error bars are an indication of how much variation is present.
So, the same graph with error bars might look like this:
That’s not too bad when it comes to error bars.
A really important point is that they don’t overlap.
Generally speaking, this is an indication that the differences were significant.
Strictly speaking, the error bars don’t show whether there was significance or not. You have to rely on statistical tests for that. However, they are a really good indication of just what the data from the study means.
In general, the smaller the error bars are, the more accurate the results are.
Look at this one in comparison:
In this case, the error bars are really large and they overlap by a considerable margin.
This is a pretty strong indication that the differences observed in the study don’t really mean a whole lot.
In general, error bars give you a really strong indication of how accurate the results of a study were.
You can generally get better information from the statistical tests done in a paper, but looking at graphs and error bars is important if you are looking for summaries. Knowing about these bars is also important for looking at graphs online because often you won’t know anything about the statistical tests.
Generally speaking, graphs with error bars tell you a whole lot more than those without them, but even so, many graphs you find won’t include the error bars.
There are many reasons for this, including the fact that the bars can be confusing unless you understand what they mean.
However, in some cases, people choose not to include error bars because the data seems more convincing without them. This isn’t a practice you see in scientific papers because there are a lot more restrictions, but online, graphs without error bars are far more common.
Scientific studies often consider statistical significance, but the biological significance isn’t discussed as much.
Biological significance is slightly different, and it refers to whether the numbers from the study mean something biologically.
Let’s go back to the example of weight loss.
It is possible for a study to find a statistically difference in weight loss between two groups of people and for this difference to also be very small.
If a study compared a group taking a weight loss supplement to a control and found a difference of half a pound per week this would suggest that the supplement helps a little.
However, if a difference of half a pound was found over a six week period, it could be argued that the supplement really didn’t do anything.
Even if the half pound difference was significant, it is largely irrelevant.
When reading scientific papers it is always important to keep an eye on numbers, because authors will often emphasize that their results are statistically significant, even if the numbers don’t really mean that much in real life.
One issue with this is that even small differences can look very convincing on graphs.
For example, this is a graph of the same weight loss comparison, but where the differences are much smaller.
If you just look at the size of the bars, it still looks like there is a major difference. But, once you look at the values, you realize that isn’t true.
This is a really misleading way of representing the data, but you will find that some people and even some studies do that.
In reality, the data should be presented like this:
As you can see, when you use this scale, the differences don’t look like anything at all.
So, you need to take both the biological and statistical significance into consideration when looking at the outcomes of any studies. It might seem tricky at first, but this is something that becomes intuitive relatively fast.
Final Thoughts
Like the authors of blogs, the authors of scientific studies have their own motivations, and they are often convinced of their own perspectives.
Because of this, it is important not to take any scientific study (or blog) at face value, and instead, seriously consider the evidence they present and whether it means anything.
While scientific papers might be a little challenging to read, understanding them can be important for determining how valid (or not valid) current advice in health and nutrition actually is.
Personally, I rely strongly on scientific evidence because the media has a tendency to report information out of context and to make it appear that studies say something different entirely.
If any of this seems confusing, the best approach is to simply try reading scientific studies and seeing what information you can get out of them.
It is certainly an approach that takes some learning, but you do get better at it over time and there really is a lot of information to be learned.
[feather_share show=”google_plus, twitter, facebook,pinterest” hide=”reddit, linkedin, tumblr, mail”]
Hi Vince,
I’ve never even thought of doing my own thorough research by looking at scientific papers, great idea!
I certainly was unaware of Google Scholar, so thanks for bringing that to my attention.
What I like about your idea here is that you’re more likely to get somewhere near the truth with better informed facts than, like you say, through the companies who are trying to sell to us.
Do you ever find though that when you read the papers, you end up with more questions than answers, or do they generally leave you satisfied?
Cheers, Tony.
I love this article. People surfing the net, especially those looking for health advice, really have to be aware of where the information they’re reading is coming from. And being able to do an academic research on your own gives you direct access to the most reliable information. I do a lot of research for my website, it’s about health. and I use Pubmed, but it’s because I’ve got access to it through a university. Another useful source of information that seems more readable than academic journals is EurekAlert. Thank you for the other links, I;ll check them now and bookamrk them for future reference.
A very sensible piece of writing giving some sound advice to do your own research and how to go about it.
I quite agree that it is difficult to weed out those facts that have just been copied and relayed ad infinitum without any backup. Because of this, I also rely heavily on research papers.
Even so, it is still difficult to know if the researcher has an agenda for the research to go his way. Being completely impartial about the results of research must be very difficult. If you find out that there is a relationship between the researcher/s and a pharmaceutical company, this can always make you question the results, especially if the research is deemed to be ground breaking. Statins come to mind!
Great site and I can see some really useful and well researched info coming from it. Ches