With dishes that are sensitive to holding time, it's impractical to do more than two at a time. So you usually end up with a series of "beat the champion" trials between two options. It takes some extra effort, but compared to the amount of effort already expended in cooking, the marginal cost isn't that severe. If you're good at it, you basically end up with a few extra dishes to wash. Maybe you need some extra freezer or pantry space.
You have an entire lifespan to refine your recipes. But you also only have one lifespan to enjoy them, so it's good to do it efficiently.
The cooking recipe problem is the same as a lot of other things. Doing things scientifically is the best way to make things better, but you can't really trust that anyone else is not cutting corners, so any time someone shares their results, you have to be skeptical. The peer review system in place for cookbooks isn't any more trustworthy in my eyes than the one for academic chemists, and that is compounded by the fact that different people like different things.
No, I'm not.
Subjective (adj.): based on or influenced by personal feelings, tastes, or opinions
> A blinded taste test is an objective, qualitative measurement
No, the kind of blinded, ranked preference test you described is a subjective, quantitative measurement (the measure is ordinal rather than interval or ratio, but an ordinal measure is still quantitative if less powerful than other classes of quantitative measures.)
When the same taster is judging two dishes at the same time, the emotional state is the same, the taste buds are the same, the mental state is the same. The effect is the same as if you had a tasting robot programmed to output a number when you drop food into its sampling port. Most of the subjectivity is removed. But different people/tastebots have different calibration settings, so you can't compare results between tasters, or even between different times for the same taster. It's only objective if you taste steps to remove the subjectivity: blindfolds on, equal-sized portions, neutral-flavor palate-cleanser between tastes, quiet room, etc.; but if you are careful, you can then have reasonable confidence that if A > B in that subjective state, it is likely that A > B in most other subjective states, too. (Any state that caused B > A would then be considered "interesting".)
A subjective, qualitative measurement would be something like Scoville heat units. Each taster has a different sensitivity to piquant chemicals, and different rates of sensory adaptation, so multiple judges are used, and results are imprecise.
An ordinal ranking is not quantitative. You can't tell the following apart:
qualitative ordinal ranking
A > B > C > D > E > F > G > H
|<-theoretic quantitative measurement->|
----A---B--C-------D-EF---G------H------
------AB----C----D----E----F-----G--H---
A------------------------------BCD--EFGH
-------------A-B-C-D-E-F-G-H------------
Because I've found the information from his shows has been useful in improving the output of my own cooking. Otherwise, I wouldn't have bought any of his cookbooks.
> If you don't know anything about cooking, you have to trust someone to tell you what to do.
Cookbooks are generally not written for, or primarily marketed to, people who don't know anything about cooking.
> So you need a curator for your curators--a reviewer for your reviewers.
That's what acquaintances you know who have demonstrated proficiency in the kitchen are for.
> Pick a set of recipes for the same dish, make all of them, have an accomplice label each with a letter--so you don't know which is which--and do a blind taste test. Rank the lettered portions according to your own preference, and then your accomplice will let you know which recipe you chose as the best.
This is optimal, but an extreme time and resource hog if you are going to do it with a wide enough range of recipes to be meaningful with no prior filter, and also logistically problematic because many dishes are sensitive to holding time (and because of variability in even skilled execution, you need to execute each recipe multiple times, grade them all, and do statistical aggregation of the results), so you need lots of parallel cooks and parallel kitchens to effectively execute this.
> Trust the objective measurements.
Ranking according to your preference is subjective measurement.
> A little bit of science in your kitchen obviates all need for unearned reputation and trust in trademarks.
If you ignore time and resource constraints, it avoids the need for curation (whether earned or not), in practice, to be usefully used, you need a curated filter in front of the process (but then, it also feeds back into the process of meta-curation by providing information on the reliability of curators.)