News for hackers. Stuff that matters.
Re:LSD microdosing: a randomized, blind self-experiment by Someone on Thursday October 17, 2013 @02:02PM (#6567037)
I also find the methodology lacking.

"This suggests the following design: a randomized dose on day 1, followed by days 2 and 3 off, then on day 4, drinking the second container; on day 7, examine the containers recording whether active/placebo and finally, starting over as day 1 of a new pair of 3-day blocks."

That means that, during the experiment, the subject can learn correlations between the test and the control ("So, I tend to go to the bathroom in the middle of the night in the test.")

If the subject has some (conscious or unconscious) prejudice for the result, he can use that information to adjust his self-reporting (again, consciously or unconsciously). For example, a subject may report a silly thought as a creative one instead of as a stupid one.

Also, given that the period of the experiment is exactly a week, I also would control for 'LSD is the first thing exactly as often as the test"

Re:LSD microdosing: a randomized, blind self-experiment by gwern on Thursday October 17, 2013 @05:07PM (#6568126)
>For example, a subject may report a silly thought as a creative one instead of as a stupid one.

You would have to assume that I was subconsciously also lying in the predictions, to hide my foreknowledge, and how would I be able to precisely affect the sleep or Mnemosyne scores?

> Also, given that the period of the experiment is exactly a week, I also would control for 'LSD is the first thing exactly as often as the test"

? 3-day blocks imply a 6-day cycle, not 'exactly a week'. And I'm not sure what you mean by 'control' either.

Re:LSD microdosing: a randomized, blind self-experiment by Someone on Thursday October 17, 2013 @05:32PM (#6568277)
I must have interpreted the "finally" in "on day 7, examine the containers recording whether active/placebo and finally, starting over as day 1 of a new pair of 3-day blocks."

And control: see http://thesocietypages.org/socimages/2013/01/12/what-does-it...

Re:LSD microdosing: a randomized, blind self-experiment by gwern on Thursday October 17, 2013 @11:26PM (#6569598)
> And control: see http://thesocietypages.org/socimages/2013/01/12/what-does-it....

Yes, I'm aware of the statistical concept, and do controlling in a few other essays. My point was: 'control' is not an instantly obvious application. What am I controlling and how? Am I modeling it as a separate additive categorical variable? Quantitative variable? An interaction? (All different in R's formula notation, because they're different - +, *, :, and |) When you say

> I also would control for 'LSD is the first thing exactly as often as the test"

I don't even know what I would be controlling here, since I don't know what the order is supposed, what the 'test' is, and so on. You may think your suggestion is clear, but to me it's as clear as mud.

Re:LSD microdosing: a randomized, blind self-experiment by Someone on Friday October 18, 2013 @01:32PM (#6572626)
If your cycle is 7 days (which it isn't, but which I initially understood it to be), you will want to make sure that, with an experiment running for 2N weeks, you take the LSD-spiked water exactly N times on Monday and N times on Thursday, rather than relying on luck to get close to that.

One way to do that is to make 2N packages containing a spiked and a non-spiked sample, with one sample marked 'take me first', and, in N cases, have the LSD-spiked sample be that sample. Of course, if you do that, it is even more important to not inform the subject about the history of he experiment before it is over.

Re:LSD microdosing: a randomized, blind self-experiment by gwern on Friday October 18, 2013 @08:35PM (#6574817)
> If your cycle is 7 days (which it isn't, but which I initially understood it to be), you will want to make sure that, with an experiment running for 2N weeks, you take the LSD-spiked water exactly N times on Monday and N times on Thursday, rather than relying on luck to get close to that.

I see. So you were suggesting throwing in a categorical variable for the day of week and including that as a additive covariate to control for fixed-effects on days. I haven't actually run into any of my experiments depending on day of week to any important degree, but I still try to avoid 7-day blocks to make that less of an issue. It's not like there's usually an advantage to running in a weekly block, so better safe than sorry.

Re:LSD microdosing: a randomized, blind self-experiment by javajosh on Thursday October 17, 2013 @02:14PM (#6567135)
Given the fact that he had a null result, I don't think this criticism could possibly be valid.
Re:LSD microdosing: a randomized, blind self-experiment by Someone on Thursday October 17, 2013 @05:24PM (#6568226)
Most importantly, the criticism of a _design_ is independent of the results of the experiment.

Secondly, he may have wanted a null result.

Re:LSD microdosing: a randomized, blind self-experiment by gwern on Thursday October 17, 2013 @11:31PM (#6569623)
> Most importantly, the criticism of a _design_ is independent of the results of the experiment.

Flaws can go in any direction, and make a design conservative or optimistic. In this case, the identified flaw does not undermine the actual result of the experiment - though it would have undermined the opposite (hypothetical) result of positive effects.

Re:LSD microdosing: a randomized, blind self-experiment by Someone on Friday October 18, 2013 @07:01PM (#6574526)
If you design an experiment in such a way that outcome A can be explained away by "yes, but the design is flawed", while outcome B cannot, I don't think it is fair to do the experiment.
Re:LSD microdosing: a randomized, blind self-experiment by gwern on Friday October 18, 2013 @08:25PM (#6574793)
Experiments don't have to be 'fair'; your 'fairness' is not required for tests to be informative. Imagine an IQ test which spits out a binary yes/no result based on whether you're in the top 10% or not. This is a useful, meaningful test, which is 'unfair' in your naive sense. Issues of information content, discrimination, false positives and negatives - it's more complex than saying 'it is fair to do the experiment'. In this case, I got a reliable negative, rather than a less reliable positive.