Everybody in the industry thought that their black-boxes were uncorrelated and independent. That was part of the appeal of them (and also why it's so difficult to sell algorithmic trading products - most firms want to develop them in-house). Problem was that they had no idea what everyone else was doing, and so most of them independently invented algorithms that behave fairly similarly.

It's like the thousands of entrepreneurs that saw Friendster and said "Oh look, only one competitor in the market, I'll build myself a social network and get rich!" Or the thousands of entrepreneurs that saw Reddit and Digg and said "Oh look, only two competitors in the market, I'll build myself a social bookmarking site and get rich!" Or the thousands of entrepreneurs that saw YouTube and said "Oh look, only one competitor in the market, I'll build myself a video sharing site and get rich!"

The only way to get rich is to build things where there are zero popular competitors in the market, or where there are 1-2 competitors and everyone believes it's fruitless to compete with them. Otherwise, you can bet that thousands of people you haven't heard of are doing the same thing, and by the time you get a product out they'll be getting a product out too.

(Incidentally, the first few quant hedge funds, like Renaissance or D.E. Shaw, made out like bandits.)

"Everybody in the industry thought that their black-boxes were uncorrelated and independent."

I'm not so sure. I interviewed at a Wall Street company that was programming data channels headed into the black-boxes. They boasted that 30% of trades were dealt with by computers. I mentioned that that should probably be enough to drive some pretty serious feedback effects. I asked: in principle, if these black-boxes handled all the trades, where would the decision to trade or not come from.

Instead of a blank stare, I was presented a cold, nervous look. He said "basically all these algorithms have been worked out in the 70's and 80's, in Academia. Nobody really knows what they're doing, or why they're doing it. But they are trying to compete by getting more data faster than the next guy. If everyone trades the same, as long as you trade faster, you'll win."

I asked if it was moral to play such a risky game with the economy.

I didn't get the job.

Everybody knows it in the sense that a debt collector knows that they're ruining people's lives, but it's kinda swept under the rug. We also had the experience of people saying "yeah, we don't really know how well our algorithms our doing, so if you can develop software to tell us, we'll pay you big bucks." (Though, last I heard, nobody had actually paid big bucks for the software we did develop.) But it's the kind of truth that nobody really wants to face, so they just kinda acknowledge it and then go about their business.

The only problem is that these funds don't start out trying to be correlated with each other--the most successful strategies are those that no one else is doing, since you, tautologically, get the best prices on broadly undervalued assets. They don't want to hold all the same things, but finance is a small world at people catch on to successful strategies pretty quickly. It just pans out that way since they are built on analyzing historical relationships between assets and what has worked in the past, e.g., buy refiners when the crack spread widens, or high p/e/g ratio stocks. To be independent you'd have to either find that relationship no one knows about, or pick a strategy that hasn't worked in the past.

To put it in web terms, it's almost like these quant funds are just adapting to industry best practices. We see that people like social networking, crowdsourcing, "web 2.0" page layouts, etc., so we see lots of sites racing to add these features--because hey, that's what works, that's what users want. But what happens if people get sick of one of those features (or any other you pick)? The relationship between the market and that feature breaks down. All those sites who counted on that strategy will all fail (or adapt) at around the same time, and many will rush into the next hot area (Pointcast-style "push" technology, anyone?).

Quant funds are basically just advanced machine-learners; you could implement a black box-of-sorts on your website by looking at Comscore numbers/trends for different sites and plotting that out against the features/layouts/topics they use, and instantly adding some new widget to your sidebar or something when you see a positive relationship with traffic generation. (Techmeme is a great example of a web black box, BTW, always on the hottest tech trend.)