News for hackers. Stuff that matters.
Re:New High I/O EC2 Instance Type - hi1.4xlarge - 2 TB of SS... by cperciva on Thursday July 19, 2012 @03:36AM (#4264914)
What sort of map-reduce space needs fast random I/O? The only things which come to mind are disguised table joins, where the right solution is to use sorting instead.

(Not trying to be argumentative, I'm just trying to figure out what you have in mind.)

Re:New High I/O EC2 Instance Type - hi1.4xlarge - 2 TB of SS... by jedberg on Thursday July 19, 2012 @03:46AM (#4264934)
It's not just random I/O that's fast, but sequential is a lot faster too.

But what I'm thinking of is the tokenization of a large corpus of text into bigrams and trigrams for example.

Re:New High I/O EC2 Instance Type - hi1.4xlarge - 2 TB of SS... by cperciva on Thursday July 19, 2012 @04:22AM (#4265022)
Good point, with all the focus on random I/O in the announcement I was forgetting about the sequential I/O performance benefit.

Hmm, that reminds me, I have an interesting toy problem which is sequential I/O limited...

Re:New High I/O EC2 Instance Type - hi1.4xlarge - 2 TB of SS... by bravura on Thursday July 19, 2012 @03:53AM (#4264953)
Admittedly, it would be fun to pull in the Common Crawl and process it.

Note that there's a code contest for Common Crawl that recently was announced: http://commoncrawl.org/first-ever-code-contest/