A p-value is a measure of how likely it is that a result is due to chance rather than a real effect. A p-value of 0.5 means that the result is 50% likely to be due to chance and the true effect exists.
A common misconception about p-values is that if it is below a certain threshold, for example, 0.05, then the result is significant and the difference is true. This is not the case!
P-values can be reported in two different ways: one way is relative to a z-score, and the other way is as a probability relative to a fixed probability such as 0.05. The latter form of p-value is what most people think of when they hear the term, but both can be different for the same results.
This article will discuss how these two types of p-values interact and contrast them with each other.
Identify the significance level
Before you can conclude that the difference between groups is significant, you have to decide what level of difference is considered significant.
This is called the significance level, and it is usually 0.05 or 0.01. The lower the significance level, the more confident you can be in your conclusion that the difference between groups is significant.
So how do you determine which significance level to use? It depends on the situation, but usually require some experience with the topic and/or research standards.
For example, in clinical settings, a 0.05 significance level is typically accepted because it prevents overdiagnosis (i.e., diagnosing a condition that does not cause any symptoms or disease progression) and unnecessary treatment (i.e., treating a condition that will go away on its own).
Identify the direction of the test
In this case, you would need to determine if the mean number of hours slept was greater than eight hours. Given that the P-value is less than 0.05, you can conclude that there is not enough evidence to say that the mean number of hours people sleep is less than eight hours.
The same would be said if the P-value was greater than 0.05. You could not say that the mean number of hours people sleep is more than eight hours.
For example, if 1,000 people were surveyed and the average amount of sleep they get was seven hours and twenty minutes, then within 95% confidence, the average amount of sleep received does not change by more than two minutes either way.
Calculate your test statistic
The test statistic is the ratio of the difference between the two groups and the standard deviation of the distribution. In other words, it is the difference between the mean BMI of overweight people and average BMI of healthy people divided by the standard deviation of overweight people.
A larger test statistic indicates that there is a greater difference between groups and therefore a higher probability that this difference is not due to chance. A smaller test statistic indicates that there is less of a difference between groups and therefore a lower probability that this difference is not due to chance.
In our hypothetical scenario, you calculated your test statistic and found that it was 1.6. This means that on average, overweight people have 1.6 more BMI points than healthy people.
Determine the critical values
Once you have your p-value, you need to determine what levels of significance the p-value qualifies as.
These levels are called confidence intervals, and they are determined by researchers as part of their A/B tests. These confidence intervals are 95%, 98%, and 99% confidence intervals.
The name of the interval depends on what percentage of the time your test will reveal a difference that is not actually there. For example, if you have a 95% confidence interval, then once you run your test, there is a 5% chance that there was no difference between the groups and that your conclusion was wrong.
Interpret your p-value
Once you’ve calculated your p-value, you must interpret it. The p-value tells you the probability of obtaining a result as significant as the one you obtained if the null hypothesis is true.
Given a 0.016 p-value, you can say that if the null hypothesis is true, then 1 – 0.016 = 0.994, or 94%, of the time you would obtain a result as significant as the one you obtained.
That seems like a low probability, but remember that the null hypothesis is that there is no difference between the two groups. So even though there is a small difference between the groups, it is supposed to be that way because of what the null hypothesis says.
If you have strong evidence against the null hypothesis, then your p-value will be lower than 1–0.014=0.986 or 0.014.
What does a p-value of 0.016 indicate?
A p-value of 0.016 indicates that the difference between the average score for the treatment group and the control group is statistically significant.
That is, it is unlikely that the difference between groups is due to chance. The p-value tells you how unlikely this is, i.e., it tells you what level of significance you have established.
A lower p-value indicates a greater level of significance, and thus a greater likelihood that there is a real difference between groups. A p-value of 0.001, for example, indicates a very strong level of significance.
Therefore, in this experiment, you can be fairly confident (95% confidence) that there is a real difference between groups due to the fact that your p-value was less than 0.05.
What does a p-value of 0.016 not indicate?
A p-value of 0.016 does not indicate that the null hypothesis is true. A p-value of 0.016 indicates that the observed data is very unlikely to have occurred by chance if the null hypothesis is true.
Put another way, it indicates that if the null hypothesis were true, we would expect to observe data as unusual as what we actually observed 0.016% of the time.
That’s a pretty low probability! So, if your p-value is equal to 0.016, you can confidently reject the null hypothesis and assume that your observation is not due to chance.
However, in science, we need to be able to replicate our results in order to confirm them. That is why there is another step involved in the scientific method: Try to replicate your results by conducting a new study using a new sample set.
Conducting multiple tests and controlling the family-wise error rate
Another issue with p-values is that researchers may be tempted to conduct multiple tests, such as comparing several treatments to each other or comparing a treatment to a control group.
This is problematic because it increases the chance of finding a false positive—finding a significant difference when there is actually no difference.
For example, if a researcher compares their treatment to two other treatments, then they are performing three comparisons. The p-value for each comparison could be below 0.05, but if they are all combined, then the family-wise error rate could be 0.05, and they would not notice this until after the results are published.
This is why researchers now publish what is called the family-wise error rate, which controls how many comparisons are made and what the probability of finding a false positive result is for all of these comparisons combined.
Leave a Reply