Re:Introducing Streaming K-Means in Spark MLlib 1.2
by michaelmior on Wednesday January 28, 2015 @02:41PM
(#8962019)
Is it true that this doesn't support dynamic values of k? That is, the algorithm isn't adaptive to a changing number of clusters? That said, I suppose for some small range of k values, you could do this trivially by tracking them all and picking the best.
[ Parent
| on HN ]
Re:Introducing Streaming K-Means in Spark MLlib 1.2
by mengxr on Wednesday January 28, 2015 @03:14PM
(#8962277)
Now we only detect dying clusters while keeping k constant. Dynamic values of k would be a nice feature to add in later releases:)
Re:Introducing Streaming K-Means in Spark MLlib 1.2
by ramses0 on Wednesday January 28, 2015 @03:38PM
(#8962412)
This is a common question for me: how to determine dynamically the number of clusters (including the splitting / merging). I've looked into jenks breaks, but it also seems to require a cluster number.
http://en.wikipedia.org/wiki/Jenks_natural_breaks_optimizati...
Do you have any advice or ideas for automatically picking count of clusters in an unknown data set?
Re:Introducing Streaming K-Means in Spark MLlib 1.2
by shazeline on Wednesday January 28, 2015 @05:39PM
(#8963019)
One common approach is to look for the elbow in the curve <metric> vs K (number of clusters). This is essentially finding the number of clusters after which the rate of information gained/variance explained/<metric> slows. I believe it's possible to binary search for this point if you can assume the curve is convex.
Re:Introducing Streaming K-Means in Spark MLlib 1.2
by tlarkworthy on Wednesday January 28, 2015 @03:37PM
(#8962402)
a general drawback of k-means. You have to select the k. In practice you try a range and see how well they summarize your data (e.g. leave one out cross validation).
You could do that here too, just have a range of k. If only there were a streaming leave-one-out cross validation for k-means to complement this approach ...
(it is possible to do this in a streaming style, see LWPR)
Re:Introducing Streaming K-Means in Spark MLlib 1.2
by michaelmior on Wednesday January 28, 2015 @03:55PM
(#8962495)
I understand that's a drawback of k-means. I was just wondering if this was something that Spark solved natively. Looks like the answer is no for the time being. Thanks for the pointer to LWPR :)
Re:Introducing Streaming K-Means in Spark MLlib 1.2
by rxin on Wednesday January 28, 2015 @03:58PM
(#8962508)
Maybe using BP-Means: http://arxiv.org/abs/1212.2126
K-Means without preset K.