Re:The Best Free and Open Source Data Mining Software
by tibbon on Wednesday December 01, 2010 @09:29AM
(#1957963)
If I may plug the 140kit (http://140kit.com) which is an open source Twitter mining and analytics solution that uses the streaming API to make downloading millions of Tweets in easily accessible formats quick and easy for anyone.
(doh, it looks like its down for some reason. Well, when it's back up- its there. I'm sure you could find the github repo and run it yourself since it is open source). Written in Rails.
Re:The Best Free and Open Source Data Mining Software
by kanak on Wednesday December 01, 2010 @10:00AM
(#1958059)
I guess R is a bit more DIY than these frameworks, but it has a very large collection of tools. I've found libraries for everything from CART (classification and regression trees), to SVM, to HMM learning, to clustering, to EM. R with libraries from CRAN is my go-to tool for statistical learning.
Re:The Best Free and Open Source Data Mining Software
by ericac on Wednesday December 01, 2010 @11:44AM
(#1958449)
R is great at 100s of thousands but fails miserably at millions. This seems especially true for clustering and regression. With the explosion of data collection, tools have to be able to easily take in and clean millions of records quickly.
Does anybody have experience with Orange or RapidMiner?
Re:The Best Free and Open Source Data Mining Software
by ez77 on Wednesday December 01, 2010 @11:07AM
(#1958293)
Something like SAS or DAP would be better suited for large data sets, as R tends to load everything on the RAM.
From http://www.gnu.org/software/dap/ : "Because Dap processes files one line at a time, rather than reading entire files into memory, it can be, and has been, used on data sets that have very many lines and/or very many variables."
Re:The Best Free and Open Source Data Mining Software
by bkj123 on Wednesday December 01, 2010 @01:30PM
(#1958903)
Regarding R's processing power, I haven't found it to be an issue. When building a model and testing, I use a sample of the data which is usually less than 100,000 observations. I use samples even when using a tool like SAS Enterprise Miner.
As far as scoring, I usually export using PMML and run it natively on the database. Makes for fast execution. PMML is available in R, RapidMiner, and other packages.
Re:The Best Free and Open Source Data Mining Software
by kanak on Wednesday December 01, 2010 @11:44AM
(#1958454)
This is definitely a problem with R, although the biggest problem IMO is that a lot of libraries aren't multicore capable. Fixing the memory problem was just a matter of adding lots of ram into our workstation, we can't fix the "can't use more than one core at a time" problem as easily.
Re:The Best Free and Open Source Data Mining Software
by nivertech on Wednesday December 01, 2010 @12:29PM
(#1958637)
Yes you can: just run it in many single-core VMs. This what I was recommended by vendor making legacy single-threaded software. They were the best in their field, so they never really tried to port it to multicore ;(
Re:The Best Free and Open Source Data Mining Software
by bluedevil2k on Wednesday December 01, 2010 @04:32PM
(#1959720)
Weka isn't limited to the GUI if you want to mine your data. It's a regular JAR file you can drop in your server-side web application and make calls to it like any other Java library. I've used it in some of my apps for some clustering algorithms (the easy stuff, since it can get complicated).
I've also written a few articles on Weka if you want to read a few nice tutorials on how to use it. I'm not a Data Mining Expert, but I've had a few semesters of it in grad school.
Re:The Best Free and Open Source Data Mining Software
by Swizec on Wednesday December 01, 2010 @11:22AM
(#1958344)
How cool is that! I'm studying Computer Science at the University that makes Orange (the first on the list). And the professor who originally came up with it is an advisor for my startup.
Re:The Best Free and Open Source Data Mining Software
by ams6110 on Wednesday December 01, 2010 @12:38PM
(#1958686)
What is your professor advising you on, specifically? I'm curious because the computer science profs I know would not have the first clue about running a startup. Not that there's anything wrong with that...
Re:The Best Free and Open Source Data Mining Software
by Swizec on Wednesday December 01, 2010 @01:22PM
(#1958857)
Actually a lot of my startup is based on the idea that we can create the machine-learning/data-mining algorithms. He's the lead of the faculty department that deals primarily with data mining.
Re:The Best Free and Open Source Data Mining Software
by uxp on Wednesday December 01, 2010 @11:11AM
(#1958312)
The Royalty Free version is open source, however it appears to be a highly restricted version of the GPL v1 license, where any application that uses lingpipe, by API calls or even by separately using the output of lingpipe, must comply with an Open Source Initiative license.
Somewhat interesting clause that may restrict "freedom", but none-the-less still enables it to be open-source.
Re:The Best Free and Open Source Data Mining Software
by NHQ on Wednesday December 01, 2010 @10:46PM
(#1960916)
google bought a company within the last couple years that had made a really smart open source data app that ran in the browser or something. Anybody know what it was called?
(doh, it looks like its down for some reason. Well, when it's back up- its there. I'm sure you could find the github repo and run it yourself since it is open source). Written in Rails.
Does anybody have experience with Orange or RapidMiner?
From http://www.gnu.org/software/dap/ : "Because Dap processes files one line at a time, rather than reading entire files into memory, it can be, and has been, used on data sets that have very many lines and/or very many variables."
As far as scoring, I usually export using PMML and run it natively on the database. Makes for fast execution. PMML is available in R, RapidMiner, and other packages.
I've also written a few articles on Weka if you want to read a few nice tutorials on how to use it. I'm not a Data Mining Expert, but I've had a few semesters of it in grad school.
http://www.ibm.com/developerworks/opensource/library/os-weka...
https://www.ibm.com/developerworks/opensource/library/os-wek...
https://www.ibm.com/developerworks/opensource/library/os-wek...
So essentially, he's advising on algorithm stuff.
http://gate.ac.uk/
http://mallet.cs.umass.edu/download.php
http://alias-i.com/lingpipe/index.html
http://incubator.apache.org/uima/
http://elefant.developer.nicta.com.au/
Somewhat interesting clause that may restrict "freedom", but none-the-less still enables it to be open-source.
http://alias-i.com/lingpipe/licenses/lingpipe-license-1.txt
http://chart.io
http://code.google.com/p/google-refine/