Re:Introducing Streaming K-Means in Spark MLlib 1.2
by mengxr on Wednesday January 28, 2015 @03:14PM
(#8962277)
Now we only detect dying clusters while keeping k constant. Dynamic values of k would be a nice feature to add in later releases:)
[ Parent
| on HN ]
Re:Introducing Streaming K-Means in Spark MLlib 1.2
by ramses0 on Wednesday January 28, 2015 @03:38PM
(#8962412)
This is a common question for me: how to determine dynamically the number of clusters (including the splitting / merging). I've looked into jenks breaks, but it also seems to require a cluster number.
http://en.wikipedia.org/wiki/Jenks_natural_breaks_optimizati...
Do you have any advice or ideas for automatically picking count of clusters in an unknown data set?
Re:Introducing Streaming K-Means in Spark MLlib 1.2
by shazeline on Wednesday January 28, 2015 @05:39PM
(#8963019)
One common approach is to look for the elbow in the curve <metric> vs K (number of clusters). This is essentially finding the number of clusters after which the rate of information gained/variance explained/<metric> slows. I believe it's possible to binary search for this point if you can assume the curve is convex.