Re:The Best Free and Open Source Data Mining Software
by bkj123 on Wednesday December 01, 2010 @01:30PM
(#1958903)
Regarding R's processing power, I haven't found it to be an issue. When building a model and testing, I use a sample of the data which is usually less than 100,000 observations. I use samples even when using a tool like SAS Enterprise Miner.
As far as scoring, I usually export using PMML and run it natively on the database. Makes for fast execution. PMML is available in R, RapidMiner, and other packages.
Re:The Best Free and Open Source Data Mining Software
by kanak on Wednesday December 01, 2010 @11:44AM
(#1958454)
This is definitely a problem with R, although the biggest problem IMO is that a lot of libraries aren't multicore capable. Fixing the memory problem was just a matter of adding lots of ram into our workstation, we can't fix the "can't use more than one core at a time" problem as easily.
Re:The Best Free and Open Source Data Mining Software
by nivertech on Wednesday December 01, 2010 @12:29PM
(#1958637)
Yes you can: just run it in many single-core VMs. This what I was recommended by vendor making legacy single-threaded software. They were the best in their field, so they never really tried to port it to multicore ;(
From http://www.gnu.org/software/dap/ : "Because Dap processes files one line at a time, rather than reading entire files into memory, it can be, and has been, used on data sets that have very many lines and/or very many variables."