News for hackers. Stuff that matters.
Re:Allstate compares SAS, Hadoop and R for Big-Data Insuranc... by absherwin on Friday January 25, 2013 @11:43AM (#5116294)
I did a similar analysis several years ago. The two fastest solutions that didn't involve writing lower-level code aren't mentioned: A specialized program targeted at the insurance industry called Emblem and Stata. If I recall, they ran as fast on a single system as RevoScale does on a cluster. Of course, comparing different benchmarks is fraught with peril.

The SAS performance reported is surprisingly bad and likely is a result of the data processing steps involved. A SAS procedure is much more similar to a program than a function. While SAS wasn't the fastest solution, most of its slowness was due to building the design matrix rather than running the actual regression. There are a number of options for both tuning how this is done and caching this preprocessing. The biggest different would be using SSD's on the machine which seems unlikely given the results.

Re:Allstate compares SAS, Hadoop and R for Big-Data Insuranc... by dbecker on Saturday January 26, 2013 @12:43PM (#5121028)
Stata won't work on data that don't fit in memory. It is fast, but it isn't an option for very large datasets.
Re:Allstate compares SAS, Hadoop and R for Big-Data Insuranc... by absherwin on Sunday January 27, 2013 @12:37AM (#5123230)
True but 150MM records with 70 degrees of freedom would likely be doable with 16GB of RAM and almost certainly with 32 as most of the variables are likely boolean and so can be stored in a single byte which Stata supports unlike R and SAS.