I'm not talking about MapReduce but rather the back end presupposed by things like MapReduce. Let me try an example. Imagine some large dataset. All the stock trades ever, say. Or all the temperature readings ever. (Maybe these particular datasets don't exist, but some do.) Suppose some Saturday afternoon a question pops into your head. Wouldn't it be great to know, say, the average, max, and min temperatures in that dataset, maybe broken down across geographical buckets. How would you go about it? The hard part isn't computing averages - the difference between computing "average" and computing "identity" is practically negligible. The hard part is obtaining the data, arranging it in a suitable form, and then being able to get at it. You want the data to already exist in zillions of morsels on lots of different computers, have a fast way of discovering which morsels your computation needs, and a fast way of accessing those morsels. Ideally there would be many copies of each morsel, giving you a lot of choices about whom to talk to.

In other words, you want a decentralized, redundant morsel discoverer-replicator. To me this sounds like what BitTorrent already does with, say, movies. Any popular movie is already split into morsels and replicated a zillion times, so any particular download (computation, in our case) can immediately start running. Of course that's only true for files that are already well-seeded. But it's easy for anyone to come along with a new movie (dataset, in our case) and plug it in, making it easy for anyone else to download (compute with), and getting faster the more popular it becomes.

(MapReduce can do the computation easily... if you've first arranged all the data appropriately. But that "if" is what, in Carmack's words, "precludes most computation of opportunity.")

The reason I wonder if BitTorrent would be suitable for solving the hard part of this problem (discovering and transferring big data) is that it already has solved it, massively. What else would you need to get true ad hoc computation on arbitrary data? A query language, an evaluator for that language that knows how to compute with morsels and assemble the results (there's your MapReduce)... and what else?

Nice ideas. It sounds to me like you should be doing a Phd.

I work in a related area, which is why the idea occurred to me. To combine the two would be awesome, but is a long-term prospect.

One more point. You mentioned BOINC. I believe BOINC sends all input data to the client with each request. In other words, it's only good for computationally intensive tasks (lots of computation per unit of data). The same is true of GPGPU. That's why I brought up "average" as an example: it's a commonplace function that is not computionally intensive at all. The work is almost entirely in finding the data and accessing it.

What I really want to know, though, is who else has thought about using BitTorrent for this kind of thing, and what they think.

Your description reminds me of cassandra, though it is not the answer to your question.