Every test compares two hypotheses:
- H0, the null hypothesis: there is no difference. This is the default hypothesis—the one we seek to refute.
- H1, the alternative hypothesis: there is a difference.
The test calculates the p-value : the probability of observing a difference at least as large as yours if H0 were true. A low p-value makes it unlikely that the result is due to chance, and we reject H0.
The alpha risk is the threshold you set for yourself, traditionally 5:1 or 3:1. This is the risk of reporting a difference that doesn't exist.
Ellistat Data Analysis classifies the conclusion into four levels, using a color-coding system:
| p-value | Conclusion |
| p ≤ 0.01 | Very significant |
| 0.01 < p ≤ 0.05 | Significant |
| 0.05 < p ≤ 0.1 | Significant limit |
| p > 0.1 | Not significant |
⚠️ «Not significant» does not mean "equal.". This means that your data isn't sufficient to draw a conclusion. With five measurements per group, a truly significant difference may still be statistically insignificant; the test doesn't have the power to detect it. That is precisely what the input is for Power.
💡 Conversely, a tiny difference becomes significant when dealing with very large sample sizes. "Significant" does not mean "important.". Always consider the magnitude of the difference in addition to the p-value: the statistic tells us whether the difference is real, while the engineer determines whether it matters.