What I meant by "experts" is people who could look at your site and give it a quick lookover according to their specialty. Web software engineers might look at a site and see code that doesn't validate, or AJAX that could be optimized a bit; UI designers might suggest that you reorder some of the buttons on your page for better ease of use, etc.
- I suggest that by being an "expert" (whatever that means) you would have too much 'domain knowledge' to know what would work best from the average internet user.
- sounds like the biased sample fallacy http://www.nizkor.org/features/fallacies/biased-sample.html
- I think best idea would be for a/b split testing perhaps using the 'army' to configure your different tests..