I would have taken a well-calibrated photo of the cup each time without looking, maybe with a color card in the bottom, and only entered results at the end of the trial.

Given that there is no documentation of whether the events during the hour of test time were more or less stressful than those before it, and no taking the time of day, diet and exercise, sleep, location (quiet island or next to a construction site), etc into account, the data seems useless.

As a note, I have no idea why he bothered trying to guess what he had taken. What possible value could that have in this type of experiment?

Perhaps the correct course of action would be to ask for feedback in the design phase of an N=1 trial, especially a longer one, to avoid some basic mistakes.

> As a note, I have no idea why he bothered trying to guess what he had taken. What possible value could that have in this type of experiment?

It gives a hint of how well the blinding worked.

Yes, that is a good point. But it also causes him to continuously think about indicators to determine what he has taken. He is constantly trying to punch the veil on this "blind" study instead of doing everything he can to avoid that. Are they really the exact same weight? Do they really feel the same in his hand, mouth and throat? Does one have a slightly different taste - after all, no filling process will leave the outside of the capsule 100% free of powder, etc? Very subtle differences could manifest themselves over 16 months.

If your sample is blinded yet you consistently guess correctly, then presumably either 1. you failed at blinding or 2. there is a strong and discernible effect, regardless of what your other metrics might say (your other metrics could always be flawed after all).

> no documentation of whether the events during the hour of test time were more or less stressful than those before it, and no taking the time of day, diet and exercise, sleep, location

Assuming the blinded samples are uniformly randomly distributed, and assuming the study goes on long enough, then you'd expect that stuff to average out.

But I agree, it should be recorded nonetheless. That way you can verify at the end that it did, in fact, average out as you expected. If it didn't then your data is invalid.

He has 94 data points. Not nearly enough to average out so many potentially confounding variables, and there is no way to know they would. That will be the case in almost all N=1 experiments. Perhaps he takes the pill at the onset of stress, and stress almost always tends to build afterwards. This would be a probable case for many people trying to use theanine in this way. I could never accept that we should presume it would average out, and thus I consider logging potentially confounding variables essential to a valid experiment of this type of uncontrolled experiment. Your hypothesis would be much more relevant in a larger experiment of course.

> Perhaps he takes the pill at the onset of stress, and stress almost always tends to build afterwards.

That would not be a problem regarding the averaging I referred to, although it could well pose a problem for measurement depending on how it interacted with the selected metrics.

Note that the averaging I refer to is not regarding all possible values of some metric, but rather any discrepancy in the distribution of metrics which we expected to follow the same distribution between the sample and the control.

I think maybe there's a misunderstanding? It seems that we both agree that a variety of additional variable should be logged. I was not suggesting to omit them, but rather to use discrepancies in them to detect fundamental issues with the data or study design. I would also expect larger studies to do the same where possible.

At 94 data points it is entirely possible that there would be outliers that would have averaged out for a larger N but did not. In such a scenario the presence of such outliers should then be taken to indicate a problem with the data (ie the more discrepancies you observe, the less you should trust the data).

I was mainly saying I don't understand why you were indicating we should presume or expect they would average out in an N=1 experiment. Even in much larger experiments that is not reliably the case. Science would be relatively easy if there were not a lot of noise in the real world. So, to be clear, my concern is mainly one of scale - the experiment is far too restricted to overlook this: the smaller the experiment, the more important this type of information. Perhaps you were not saying that such averaging might be possible in an N=1 experiment and I misread, since it seems that you comment here indicates a different point.

I of course agree that logging them is basic scientific methodology - in order to detect issues with the experiment, and even hopefully to see the signal through the noise.