Medical progress often starts with observations, and is confirmed by trials. Penicillin is one such example.

To discount the value of observations is to ignore an important part of the discovery process.

There is extremely limited data available around the use of nootropics for mind enhancement (as opposed to in medical use for treating illness). In the absence of of such data a survey like this indicates that there are a few things worth studying in more depth (and some supposed nootropics that seem worthless).

The author appears extremely aware of the problems with their methodology, and makes no attempts to hide these problems. To pull quotes:

The online nootropics community consists of mostly healthy individuals looking to use these substances to gain motivation or clearer thinking. They tend to “outpace” the scientific literature, using some substances that haven’t yet been shown to work, or haven’t been shown to benefit healthy individuals. ... " I would have liked to have some placebo nootropics to use as a comparison, but since I wasn’t running a trial, I didn’t have that option. I did, however, try to assess the placebo effect in a couple of sneaky ways." ... "I have very low confidence that these represent real effects rather than artifacts of the gymnastics I had to do to make the data correlatable at all, and I report them only to encourage other people to do more sober analyses."

I think attempting to do this - knowing that people will criticize the methodology - is admirable.

We can safely report the statistic that "more people self report caffeine having an impact than ginseng."

The author goes as far as noting how many people are reporting an impact where there appears to be no known mechanism that impact could be occurring. That's a defensible statistic too.

Data from self-reported surveys isn't entirely useless, and is in medical research often enough for there to be studies on the kinds of error that arise from them: http://scholar.google.com.au/scholar?q=self+reporting+survey...

Your exhortation that he should spend more time on basic understanding of statistics is misplaced, because he understands these issues, but he has different philosophical views than you on how information should be gathered and processed. Specifically, it seems to be your view that information received with noise and bias should be given (literally) zero weight, that information should be ignored unless it comes in the form of peer-reviewed, random controlled clinical trials. His philosophical view is that all information should be given some appropriate weight, and that the costs to acquire information are relevant. Random, controlled clinical trials are costly, and so we will never have them on a variety of topics of interest to us. To account for this finiteness of resources, and to suggest which areas could be more fruitful for future formal trials, it seems wise to acquire that information which can be cheaply had. This is an attempt to gather that information.

I mean, obviously there is a bias in favor of people who use more nootropics and find nootropics more effective. But since all I was doing was ranking nootropics relative to one another, that should mostly cancel out. I'm not sure why respondents would, for example, be expected to be a group disproportionately likely to have a stronger response to modafinil relative to bacopa compared to the general population.

More pressingly, I'm not sure how I can avoid it besides gathering hundreds of people together and forcing them to try nootropics one by one in controlled conditions, for which I don't have the resources. Even professional studies that select their subjects ahead of time have atrociously high dropout rates. If you have an implementable idea for avoiding response bias on next year's survey I'm happy to go ahead with it, but "not collect any survey data at all because available methodology isn't perfect" seems strictly inferior.

I feel like I hedged myself pretty appropriately: " we can't get good data without way more effort and resources than anyone is willing to put in, but if we ran a survey we could at least get better anecdata", "This survey does not intend to rectify [the confusion as to whether nootropics work at all]", "a pessimist might say that [this is] extracting patterns from random noise", et cetera. But given the choice between giving up because it wasn't perfect and going ahead with hedges, I figured I might as well go ahead.

I especially invite comments from participants here who have a professional understanding of statistical studies in clinical trials of new medicines.

Scott has an M.D. and is currently doing his residency in psychiatry.

If only the author understood this. Then he might have written this in the fifth paragraph: "They tend to “outpace” the scientific literature, using some substances that haven’t yet been shown to work...people combine a bewildering variety of chemicals without being sure which ones work better than others or even whether any of them work at all.

This survey does not intend to rectify this state of affairs"

He might also have written in the conclusion: "Overall I was not too impressed with the results of this survey. No unexpected substances jumped out as promising, there was a very wide distribution from “useless” to “life-changing” in nearly everything except the -afinils, and it was very hard to find much of a dose-response relationship anywhere."

"Other goals for the future are to drag people kicking and screaming into reading directions so I know everyone’s rating off the same scale..."

Not every blog post needs to rise to the level of double blind clinical trial.

I note that Norvig is a really high bar, and that reading and discussing articles is not going to have that level of effect either. (Though what effect it does have, comes at less risk.)

Many people claim that taking modafinil allows them to lose a night's sleep and feel fine the next day. It seems unlikely that they'd be so completely wrong that modafinil is not doing them any short-term good at all. Like, the typical response to "lose a night's sleep" is "have a really obviously bad day tomorrow". It would be difficult for modafinil to make someone so totally oblivious, to their own performance and the comments of people around them, as to not notice this happening.

If modafinil didn't have any clinical studies backing it up, the anecdotal evidence would be enough to convince me with high probability that modafinil has short-term benefits for many people.

http://www.reddit.com/r/Nootropics/comments/1xglcg/a_survey_...

This wasn't meant to convince anyone to take nootropics.