> Fractional factorial is also most effective when the covariates are orthogonal and independent, which is rarely the case. We can get around this by projecting high dimensional covariates into lower dimensional space using PCA, which guarantees orthogonality, but this also seems to not be done so much.

I guess you're right in that if there were very many interactions between variables, then the aliasing of a fractional design would be an incredible nuisance, but on the other hand if there were no dependence between any of the variables at all then there would be no point to any sort of factorial design as you could just test each variable sequentially.

> Just wondering if anyone is using fractional factorial designs in real life? (or optimal designs like D-optimal designs)

I have to admit, I have never ever seen anyone calculate a D-optimal or G-optimal design for anything -- that said, I only graduated as a statistician 6 or 7 years ago. As you say, there might be some use to it for continuous variables but for factors, given that latin squares (etc.) are known to be optimal so you can just grab that off the shelf.

I think in many industrial settings Bayesian optimisation is more practical. Since it allows use to refine our design on the fly and put resources into promising areas of the space earlier. Some kind of pre-planned fractional design could be useful for generating an initial set of points to test.

I've at times made fairly extensive use of d-optimal designs in formulation type research and development, usually in an iterative process, pruning and adding variables as well as refining ranges. Lot of footguns, but the approach can drastically decrease optimization time. In my experience it needs the right combination of: too many factors to make full factorial reasonable, too much (expected) covariance to trust single-vaeiable optimization, and cheap enough to commit to doing 8+ experiments before a major evaluation.

What are the footguns you've run into with D-optimal designs?

Mainly around bounding variables, especially when coupled with overly ambitious scope/number of tests. The short version is that its really easy to wind up with say a 80 experiment test where half or more of the combinations are invalid or perform poorly to the degree they don't adequately lend confidence to the prediction in the region of the global maximum. Leads to wasted work and the poor predictive ability doesn't lead well into focused future work. There are also the usual statistical foot guns of p-hacking and such.

For a concrete example, consider a bread baking optimization, considering time, temperature, and baking soda %. This gives a cubic design space, for which the naively (ignoring expected covariance of temperature*time) optimal design for 9 experiments is the cube's corners + the center point. If, for instance, your predetermined t_max always results in a briquette instead of a loaf, ~half of your experimental data is going to be worthless.

With more nuance, and temp_max is genuinely the highest reasonable temperature there are still two problems:

a. covariance is likely to drive combinations into 'invalid' territory (e.g. temp_max + time_max is likely to be invalid, or 2/9ths of your experimental. temp_min / time_min is also likely invalid for another 2/9ths).

b. predictive power / linearity of response over the ranges specified. Even if the combinations aren't invalid, if they are all the min/max combinations are poorly performing (due to overly wide, but valid, boundaries) you can wind up with poor predictive performance.

Covariance can be accounted for, design space trimmed (e.g. to a cube with a corner or two cut off), and bounds set conservatively but that is all tricky manual intervention that relies on knowledge of the problem domain and scaling factors of the underlying physics. When the problem isn't well understood it is easy to make errors in assumptions, those errors have a high cost, and if the problem was well understood a DOE probably wouldn't be necessary.

edit: for a less trivial example of a suitable problem for a d-optimal DOE, but with tricky bounding / underlying physics, consider: a 4 part formulation of fumed silica, cyanoacrylate, isopropanol, and water to make a gap-filling/quick-setting adhesive.

I just took this class last semester. I've been trying to find things to do with what I learned but I haven't found anything yet.

For coffee affectionados out there: picking best coffee setup settings is very similar to a physics experiment, and you can totally use the knowledge of experiment design theory there.

Settings:

* Kinds of beans at various levels of roasting available to you.

* Grind size on a grinder with controllable grind size in steps.

* Ratio of ground coffee weight to water weight used for brewing.

* For filter coffee: preinfusion time, brew time or flow rate (see V60 brewer).

* For espresso coffee: preinfusion time & pressure, brew time & pressure or flow rate (see Flair Espresso).

* Water temperature.

* Water mineral composition.

* For milk-based coffee drinks: kinds of milk, milk percentage as ratio of the total drink volume, steaming duration, final milk temperature.

All of these create a huge factorial space of possible configurations, especially if you've go into flow or pressure profiling. If you frame it as an experiment you might isolate some of the variables that make the most impact tailored to your coffee preferences.

And here is an example of a scientifically inclined paper in this space if you went to get inspired: https://www.cell.com/matter/fulltext/S2590-2385(19)30410-2?_...

Random search beats grid search!

I'm in biotech (as a software engineer/data scientist) and here this stuff is used a lot. Experiments are very hands on and expensive in terms of inputs so you want to be efficient with your use of them.

Bayesian approaches (e.g. optimization with Gaussian processes) tend to be very serial in nature: later sampling points are decided from the information you get earlier on. But these biological experiments can take days or weeks so you want to run them in parallel as much as possible. DoE is excellent for that.

I'm speccing out a D-optimal design as we speak (though I'm not sure this is what we will use for the particular project I'm on).

I used a lot of these techniques as an operations research analyst in the early 2010’s.

I also wondered the same thing about DoE for hyperparameter tuning. Always felt like it was a case of grid search/random search/bayes opt being “good enough” and easier. But maybe DoE would be worth it for something like LLMs where training runs for a month on 25k GPUs.