I've read my share of gwern reviews, and he does double-blind testing on himself and all, but there's an obvious bias that all the tests are done on gwern. Similarly, if you let people submit their own data you're still only getting data from those people, and in this case it wouldn't even be blinded.

A real experiment has access to much better populations, tendency of psych experiments to be done on psych students aside.