For decades, the same thing has been repeated in industrial statistics courses: in experimental designs, everything was invented between the works of Fisher (1926, then The Design of Experiments (in 1935) and the 1950s. The theory was considered complete.
That's no longer true. During a recent speech, Maurice Pillet, co-founder of Ellistat and a leading French authority on quality and Six Sigma, presented two approaches that are tangibly changing the way companies conduct testing: the Smart Designs and the’Bayesian optimization. Here are the key takeaways.
The Limitations of Classical Experimental Designs
Visit experimental designs Traditional methods have been immensely helpful. But they are based on assumptions that are often overlooked.
The Linearity Assumption
A two-level design (one minimum, one maximum) assumes that the response varies linearly between the two limits. This is true locally, just as the Earth appears flat when you look down at your feet. As soon as the scope of the study expands or the phenomenon is nonlinear, the model becomes misleading. It is necessary to switch to models with more levels, which leads to experimental designs with a number of trials that is often prohibitive.
Choosing the Terminals
Second pitfall: if the min and max values were chosen poorly, the optimum may lie outside the domain under consideration. The graph will then yield a valid solution… but in the wrong place.
Screening Plans and the Alias Problem
When there are many factors (six, for example), screening designs are typically used: the two-level L12 table, or the L18 table to include three levels. Their major drawback is that the effects of the factors are partially mixed (aliased) with the interactions. It is therefore impossible to know with certainty whether the observed effect stems from the factor itself or from an interaction between two factors.
Smart Design: Achieving the Key Effects—Curvature and Interactions—with Few Iterations
A set of specifications that had long been impossible to meet
What manufacturers really want is all of the following at once:
- understand the main effects clearly and unambiguously; ;
- detect any nonlinearity; ;
- identify significant interactions; ;
- combine two-level qualitative factors with three-level quantitative factors; ;
- to do as few trials as possible.
Until very recently, with six factors, there were only two options: the full design (which is unrealistic) or a response surface design with at least 54 trials. However, the full model (main effects, quadratic terms, interactions) has 28 coefficients, most of which are zero in practice. The problem is that we don’t know in advance which ones.
What Smart Design Changes
Recent scientific publications by Núñez et al. (2020) and (2023) have led to the development of a new family of designs. Their key property is that the main effects are uncorrelated with both the quadratic terms and the interactions. You are confident in your main effects, and the remaining correlations are weak enough to estimate the second-order effects.
In the example presented by Maurice Pillet (a two-level catalyst, five other three-level factors), A plan with 18 trials is sufficient where 54 were needed.
How to Analyze It
Rather than a traditional step-by-step selection process (which is impossible when there are fewer trials than coefficients), the analysis is based on finding the best subset of terms : The software tests all possible combinations and selects the most relevant model. The result reflects the actual structure of the phenomenon, sometimes including a superfluous term with a minor effect that has no practical impact on the conclusions.
To make the tool usable on a daily basis, more than 2,000 designs have been pre-calculated and integrated into the Ellistat software. The user enters their parameters, and the software suggests available designs along with their quality criteria.
Maurice Pillet's assessment is straightforward: in the corporate world, he now works almost exclusively on Smart Designs.
Bayesian Optimization: Finding the Operating Window When You Don't Know Where to Look
A Typical Example: Laser Welding
Some processes are «hit or miss»: a setting works, but change one parameter and everything goes wrong. Maurice Pillet cites the case of the same part being laser-welded by two suppliers, each using completely different settings, and neither achieving truly satisfactory results. There is probably a range within which the process works perfectly. But where?
In this case, a traditional design of experiments—which explores the corners of an already defined domain—is not appropriate.
Grid-checking? Bad idea
The natural instinct is to divide the space into a grid: 4 × 4 = 16 trials. The problem is that each value of each factor is tested four times, and a small favorable area might slip through the cracks.
Visit filling plans (space-filling designs) distribute the experiments evenly, with each experiment testing a different value for each factor. With the same number of experiments, the probability of «hitting» the right area increases significantly. But to make sure nothing is missed, one might be tempted to increase the number of trials to 100… the vast majority of which would serve no purpose.
The sequential approach: Gaussian processes + Bayesian optimization
Bayesian optimization combines two mathematical tools:
- Gaussian Processes, which model the response and, above all, the uncertainty at each point in the domain. Unlike polynomial models, this uncertainty varies from one location to another.
- Bayesian Optimization, which selects the next experiments by balancing two approaches: capitalizing on promising areas and exploring areas that are still poorly understood, where uncertainty raises hopes for a better outcome.
The process becomes sequential: about ten initial filling-plane tests, then the software suggests the next tests, explaining why. We run them, observe the convergence, and stop when the optimal region is sufficiently documented. This approach is much more in line with technicians« intuition than the »I’ll do the entire design, then analyze it” method.
To help users become familiar with the method, the Ellistat software includes a demo mode that simulates a three-factor process (temperature, pressure, cycle time).
Smart Design or Bayesian Optimization: When to Use Which?
The two approaches are not mutually exclusive; they address different situations:
- Smart Design : You have a general idea of the operating range and want to model it in detail. A maximum of three levels, with few adjustments—ideal when changing a parameter is costly.
- Bayesian Optimization : You don't know the process very well and need to figure out what works. You have to be willing to change the settings often; that's the key to exploring.
In both cases, the goal is the same: to solve increasingly complex industrial problems without having to run more tests.
Advanced mathematics, invisible to the user
Behind these methods lie complex mathematical tools. Ellistat’s approach is to make them transparent. As Maurice Pillet summarizes, the tool is not designed for statisticians, but for technicians.
Conclusion
Visit experimental designs haven’t said their last word. Smart Designs make it possible to achieve in 18 trials what previously required 54, with no ambiguity regarding the main effects. Bayesian optimization paves the way for processes that were previously adjusted by trial and error. For methods, quality, and process R&D teams, this means fewer trials, less material consumption, and more robust process settings.
FAQ
- What is a "Smart Design" in a design of experiments? This is a family of modern experimental designs that allows one to estimate the main effects without confusing them with interactions or quadratic effects, while detecting these second-order effects with very few trials.
- What is the difference between a screening plan and a Smart Design? A conventional screening design (L12, L18) partially mixes the effects of the factors with the interactions. Smart Design eliminates this ambiguity regarding the main effects and allows for two- and three-level mixed factors.
- When should you use Bayesian optimization instead of a traditional design of experiments? When you don't know where a process's operating range lies and need to explore it. If the range is already roughly known, a Smart Design is more appropriate.
- Do you have to be a statistician to use these methods? No. In the Ellistat software, the calculations are hidden: the user enters the factors, performs the suggested tests, and reads the results.
