News for hackers. Stuff that matters.
Re:Artificial intelligence: Riders on a swarm by SeanLuke on Friday August 20, 2010 @07:46PM (#1621915)
> This situation seems to have balkanized the field (page through a recent GECCO proceedings, for example), and holds back wider progress.

Stochastic optimization has always been balkanized, but not because of a lack of methodology or benchmarks, but rather because the field was simultaneously invented in several different locations (Evolutionary Programming in San Diego and at NSF, Evolution Strategies in Germany, the Genetic Algorithm in Michigan). That's what happens in such situations: you get competition and differentiation when there really isn't much. Due to various political reasons, GECCO and CEC broke apart and GECCO itself decided to divide itself up by topic. In the meantime, alternative methods (ACO, PSO, various single-state "metaheuristics", etc.) have danced about at the periphery.

I think your criticism of benchmarks is much too harsh. First off, it's not really true. A number of areas in evolutionary computation have very well established benchmarks: certainly this is the case for genetic programming; and for vector representations (Rastrigin? Rosenbrock? Schwefel? The De Jong test suite? Griewangk? Etc.). Also multiobjective optimization has established a fairly common set of benchmarks. Second, much of the remainder of the field consists of different kinds of solution representations (ACO; various graph representations such as NEAT/HyperNEAT; list representations; etc.) and in such situations unifying benchmarks make no sense. It's like insisting that the Iris data set be used for problems in text mining.

I think there's quite a large degree of theoretical work among these techniques. Indeed there are entire theoretical conferences. But if you're looking for field-wide theoretical foundation, you're barking up the wrong tree. The problem is that most "interesting" solution representations result in dynamics which are essentially impossible to base a theoretical foundation on. In a real sense, various other fields have theoretical foundations because their problems are well formed. Stochastic optimization by its nature is tackling nastier problems for which there is no well-formed solution concept. The problems are ugly and hairy. This shouldn't reflect on the researchers brave enough to tackle them.

Re:Artificial intelligence: Riders on a swarm by another on Saturday August 21, 2010 @11:58AM (#1623078)
Thanks, enjoyed your response.

Your "tackling nastier problems" point is well-taken, and one reason I have a great deal of sympathy for work in these research areas.

I think that you're painting an overly-rosy picture of the benchmark situation, but you're right in that I was likely too critical. The attention paid to synthetic fitness landscapes, for example, is also a positive sign that that many researchers do care about issues of comparison and analysis.

(That said, I obviously believe that the field can and must do much better---and, since it relies so heavily on empirical evidence, its need for consistent methodology is somewhat higher.)