News for hackers. Stuff that matters.
Re:Introducing Streaming K-Means in Spark MLlib 1.2 by michaelmior on Wednesday January 28, 2015 @02:41PM (#8962019)
Is it true that this doesn't support dynamic values of k? That is, the algorithm isn't adaptive to a changing number of clusters? That said, I suppose for some small range of k values, you could do this trivially by tracking them all and picking the best.
Re:Introducing Streaming K-Means in Spark MLlib 1.2 by mengxr on Wednesday January 28, 2015 @03:14PM (#8962277)
Now we only detect dying clusters while keeping k constant. Dynamic values of k would be a nice feature to add in later releases:)
Re:Introducing Streaming K-Means in Spark MLlib 1.2 by ramses0 on Wednesday January 28, 2015 @03:38PM (#8962412)
This is a common question for me: how to determine dynamically the number of clusters (including the splitting / merging). I've looked into jenks breaks, but it also seems to require a cluster number.

http://en.wikipedia.org/wiki/Jenks_natural_breaks_optimizati...

Do you have any advice or ideas for automatically picking count of clusters in an unknown data set?

Re:Introducing Streaming K-Means in Spark MLlib 1.2 by shazeline on Wednesday January 28, 2015 @05:39PM (#8963019)
One common approach is to look for the elbow in the curve <metric> vs K (number of clusters). This is essentially finding the number of clusters after which the rate of information gained/variance explained/<metric> slows. I believe it's possible to binary search for this point if you can assume the curve is convex.
Re:Introducing Streaming K-Means in Spark MLlib 1.2 by tlarkworthy on Wednesday January 28, 2015 @03:37PM (#8962402)
a general drawback of k-means. You have to select the k. In practice you try a range and see how well they summarize your data (e.g. leave one out cross validation).

You could do that here too, just have a range of k. If only there were a streaming leave-one-out cross validation for k-means to complement this approach ...

(it is possible to do this in a streaming style, see LWPR)

Re:Introducing Streaming K-Means in Spark MLlib 1.2 by michaelmior on Wednesday January 28, 2015 @03:55PM (#8962495)
I understand that's a drawback of k-means. I was just wondering if this was something that Spark solved natively. Looks like the answer is no for the time being. Thanks for the pointer to LWPR :)
Re:Introducing Streaming K-Means in Spark MLlib 1.2 by rxin on Wednesday January 28, 2015 @03:58PM (#8962508)
Maybe using BP-Means: http://arxiv.org/abs/1212.2126

K-Means without preset K.