News for hackers. Stuff that matters.
Re:The Best Free and Open Source Data Mining Software by kanak on Wednesday December 01, 2010 @10:00AM (#1958059)
I guess R is a bit more DIY than these frameworks, but it has a very large collection of tools. I've found libraries for everything from CART (classification and regression trees), to SVM, to HMM learning, to clustering, to EM. R with libraries from CRAN is my go-to tool for statistical learning.
Re:The Best Free and Open Source Data Mining Software by mcgin on Wednesday December 01, 2010 @10:26AM (#1958144)
I am amazed R didn't make it to the list.
Re:The Best Free and Open Source Data Mining Software by ericac on Wednesday December 01, 2010 @11:44AM (#1958449)
R is great at 100s of thousands but fails miserably at millions. This seems especially true for clustering and regression. With the explosion of data collection, tools have to be able to easily take in and clean millions of records quickly.

Does anybody have experience with Orange or RapidMiner?

Re:The Best Free and Open Source Data Mining Software by ez77 on Wednesday December 01, 2010 @11:07AM (#1958293)
Something like SAS or DAP would be better suited for large data sets, as R tends to load everything on the RAM.

From http://www.gnu.org/software/dap/ : "Because Dap processes files one line at a time, rather than reading entire files into memory, it can be, and has been, used on data sets that have very many lines and/or very many variables."

Re:The Best Free and Open Source Data Mining Software by bkj123 on Wednesday December 01, 2010 @01:30PM (#1958903)
Regarding R's processing power, I haven't found it to be an issue. When building a model and testing, I use a sample of the data which is usually less than 100,000 observations. I use samples even when using a tool like SAS Enterprise Miner.

As far as scoring, I usually export using PMML and run it natively on the database. Makes for fast execution. PMML is available in R, RapidMiner, and other packages.

Re:The Best Free and Open Source Data Mining Software by kanak on Wednesday December 01, 2010 @11:44AM (#1958454)
This is definitely a problem with R, although the biggest problem IMO is that a lot of libraries aren't multicore capable. Fixing the memory problem was just a matter of adding lots of ram into our workstation, we can't fix the "can't use more than one core at a time" problem as easily.
Re:The Best Free and Open Source Data Mining Software by nivertech on Wednesday December 01, 2010 @12:29PM (#1958637)
Yes you can: just run it in many single-core VMs. This what I was recommended by vendor making legacy single-threaded software. They were the best in their field, so they never really tried to port it to multicore ;(