Parametric and non-parametric statistical tests

Before going into the details of parametric and non-parametric tests, let's recall how a statistical test works. The Ellistat Data Analysis module allows you to perform these tests.

A statistical test works as follows:

• We consider a null hypothesis in which there is no difference between the samples.
• We calculate the probability of falling into the same configuration as that obtained with the samples observed following the null hypothesis. This probability is known as the "alpha risk" or "p-value".
• If alpha risk < 5%, it is considered too unlikely to obtain such a configuration under the null hypothesis. We therefore reject the null hypothesis and consider the difference between the samples to be significant. For this reason, all the statistical test results proposed by Ellistat will be associated with an alpha risk value on the following scale:

The number below the scale is equal to the alpha risk of the test:

1. If alpha risk < 0.01, the difference is considered highly significant<.
2. If alpha risk < 0.05, the difference is considered significant.
3. If alpha risk < 0.1, the difference will be considered borderline (we can't say there's a significant difference, but the hypothesis is interesting).
4. If the alpha risk is greater than 0.1, the difference will be considered insignificant

Example

To illustrate how a statistical test works, let's take the following example.

Suppose we want to detect whether a coin has been tipped by flipping a coin. We assume that the coin always lands on tails.

Throw n°1

After the first toss, the coin lands on tails. Can we deduce from this that the coin has been tipped?

On the face of it, it would be rather risky to bet that the part is piped, as it could very well have happened with a standard part.

In this case, the null hypothesis is: the coin is not tipped, so it has a 50/50 chance of coming up heads or tails. The probability of an unpiped coin landing on tails is 50%.

Therefore, the probability of obtaining tails after the first toss of an unpiped coin is 50%, so the alpha risk of the test is:Alpha risk = 50%

That is, there is a 50% chance of obtaining the same result by following the null hypothesis.

Throw n°2

After the second toss, the coin lands on tails again. The alpha risk becomes:Alpha risk = 25%

Does this mean that the coin is a dud?

The question then arises: at what alpha risk can we say that the coin has been tipped?

As a general rule, in industry, the alpha risk limit is set at 5%.

In other words:

• If alpha risk < 5%, the null hypothesis is rejected and the coin is considered to be tipped.
• If the alpha risk is greater than 5%, we cannot conclude that the coin is loaded. However, this does not necessarily mean that the coin is not loaded, as it depends on the number of tosses that have been made.

Example continued

Let's continue our example:

-3^ème toss, coin lands on tails: alpha risk = 12.5%

-4^ème toss, coin lands on tails: alpha risk = 6.75%

-5^ème Coin toss: Alpha risk = 3.375%

In this case, from the 5th^ème We can therefore say that the coin is tipped with a risk of less than 5%.

Parametric vs. non-parametric tests

When making population comparisons, or comparing a population with a theoretical value, there are two main types of test: parametric and non-parametric.

Parametric tests

Parametric tests work on the assumption that the data we have available follows a known type of distribution law (usually the normal law).

To calculate the alpha risk of a statistical test, simply calculate the mean and standard deviation of the sample to access the sample's distribution law.

With the distribution law perfectly known, we can calculate the alpha risk based on the theoretical calculations of the Gaussian distribution.

These tests are generally very fine, but they require that the data actually follow the assumed distribution law. In particular, they are very sensitive to outliers and are not recommended if outliers are detected.

Non-parametric tests

Non-parametric tests make no assumptions about the distribution of the data. They are based solely on the numerical properties of the samples. Here's an example of a non-parametric test:

We want to check that the median of a population is different from a theoretical value. We measure 14 pieces and obtain the following sample:

11 times on the same side out of 14

11 times out of 14, the result is below the theoretical median. If the median of the population is equal to the theoretical value, we should have 50% of coins above the median and 50% of coins below. To determine the significance of the deviation of the median from the theoretical median, it is therefore sufficient to check whether the frequency of 11 times out of 14 is significantly different from 50%.

This gap is borderline.

As in the previous example, non-parametric tests do not need to assume a particular type of distribution to calculate the alpha risk of the test. They are very elegant and based on numerical properties. What's more, they are not very sensitive to outliers and are therefore recommended in this case.

Table: Which parametric test, and which nonparametric equivalent?

Don't overlook the line representing the variation. In production, two machines can have the same average but very different levels of variation, and it is the variation that determines capability. For more information, see our article on Comparison of Distributions Using the Cramér-von Mises and Energy Tests explains these approaches in detail.

How do you choose between a parametric and a nonparametric test?

Ask yourself four questions, in this order:

1. How many samples are you comparing, and are they paired? Measuring the same parts twice (before and after a change, or using two different measurement methods) is called a paired test. Comparing different parts is called an independent test.
2. Do the data follow a normal distribution? Check this using a normality test and a histogram. A clearly skewed distribution suggests a nonparametric test.
3. Are there any outliers? If so, and if they cannot be explained by a data entry or measurement error, opt for a nonparametric test. A outlier test helps you decide.
4. What is the sample size? With a large amount of data, parametric tests become robust to moderate deviations from normality. With a small amount of data, it is difficult to verify normality, and nonparametric tests are a safer choice.

When in doubt, it is good practice to run both types of tests. If they yield the same results, the conclusion is robust. If they differ, this is a sign that the assumptions of the parametric test warrant closer examination.

"Not significant" does not mean "equal": equivalence tests

This is the most common misinterpretation in industry. A test that finds no significant difference does not prove that the two groups are identical. With only a few items, even a real discrepancy may go unnoticed.

When the goal is to demonstrate that a new supplier, a new tool, or a new setting results in the same result Unlike the old one, you have to reverse the logic and use a equivalence test. We set the maximum acceptable deviation, then try to prove that the actual deviation is smaller. The Data Analysis module actually suggests this directly when a difference test is not significant.

Another point to consider: the number of pieces needed. A calculation of power Before the measurement campaign, avoid concluding that «there is no difference» simply because the sample size was too small.

Perform these tests with Ellistat

In the module Data Analysis by Ellistat, the «Inferential Statistics Tests» view includes a «Parametric» / "Nonparametric" selector. Depending on the number of samples and whether they are paired, the software directly displays the appropriate tests, grouped by family:

• test: t-test, ANOVA, Mann-Whitney, Wilcoxon, Kruskal-Wallis, Friedman…;
• dispersion: F-test, Bartlett's test, Levene's test…;
• Distribution: Cramér-von Mises, energy test.

Each parametric test includes a pictogram that verifies its assumptions (normal distribution, outliers, equal variances). You can immediately tell whether the test is applicable to your data.

Conclusion

Choose between a a parametric test and a nonparametric test is equivalent to comparing your data with the test's assumptions:

• Normal data with no outliers: the parametric test is the most sensitive;
• Unknown, asymmetric, or extreme values: a nonparametric test is the safest choice.

In both cases, three habits can help you avoid making bad decisions:

• consider both the spread and the mean;
• Never confuse «insignificant» with «identical»;
• Determine the sample size before taking measurements.

FAQ

What is the difference between a parametric test and a nonparametric test?
A parametric test assumes that the data follow a known distribution—usually the normal distribution—and is based on the mean and standard deviation. A nonparametric test makes no assumptions about the distribution and is based on the ranks or signs of the data.

When should a nonparametric test be used?
When the data do not follow a normal distribution, when they contain outliers that cannot be excluded, or when the sample is too small to test for normality.

What is the nonparametric equivalent of Student's t-test?
For two independent samples, use the Mann-Whitney test. For two paired samples or a single sample compared to a target value, use the Wilcoxon test and the sign test.

What is the nonparametric equivalent of ANOVA?
The Kruskal-Wallis test for independent groups and the Friedman test for paired measurements.

Does a non-significant test prove that two groups are identical?
No. It simply indicates that the data do not show a difference. To prove that two groups are equivalent, an equivalence test is required, with a maximum acceptable difference defined in advance.