News for hackers. Stuff that matters.
Re:Introducing Streaming K-Means in Spark MLlib 1.2 by rxin on Wednesday January 28, 2015 @02:00PM (#8961797)
This is a cool feature, and is one of the prime example of what Spark's tight integration of various libraries can enable (in this case Spark Streaming and MLlib). It was originally designed by Jeremy Freeman to handle workloads in neuroscience, which IIRC was generating data at 1TB/30mins.
Re:Introducing Streaming K-Means in Spark MLlib 1.2 by hcrisp on Wednesday January 28, 2015 @02:46PM (#8962056)
Sounds similar to an exponential moving average[1], which itself is a one-pole IIR digital filter. [1] http://en.wikipedia.org/wiki/Moving_average#Exponential_movi...
Re:Introducing Streaming K-Means in Spark MLlib 1.2 by michaelmior on Wednesday January 28, 2015 @02:41PM (#8962019)
Is it true that this doesn't support dynamic values of k? That is, the algorithm isn't adaptive to a changing number of clusters? That said, I suppose for some small range of k values, you could do this trivially by tracking them all and picking the best.
Re:Introducing Streaming K-Means in Spark MLlib 1.2 by mengxr on Wednesday January 28, 2015 @03:14PM (#8962277)
Now we only detect dying clusters while keeping k constant. Dynamic values of k would be a nice feature to add in later releases:)
Re:Introducing Streaming K-Means in Spark MLlib 1.2 by ramses0 on Wednesday January 28, 2015 @03:38PM (#8962412)
This is a common question for me: how to determine dynamically the number of clusters (including the splitting / merging). I've looked into jenks breaks, but it also seems to require a cluster number.

http://en.wikipedia.org/wiki/Jenks_natural_breaks_optimizati...

Do you have any advice or ideas for automatically picking count of clusters in an unknown data set?

Re:Introducing Streaming K-Means in Spark MLlib 1.2 by shazeline on Wednesday January 28, 2015 @05:39PM (#8963019)
One common approach is to look for the elbow in the curve <metric> vs K (number of clusters). This is essentially finding the number of clusters after which the rate of information gained/variance explained/<metric> slows. I believe it's possible to binary search for this point if you can assume the curve is convex.
Re:Introducing Streaming K-Means in Spark MLlib 1.2 by tlarkworthy on Wednesday January 28, 2015 @03:37PM (#8962402)
a general drawback of k-means. You have to select the k. In practice you try a range and see how well they summarize your data (e.g. leave one out cross validation).

You could do that here too, just have a range of k. If only there were a streaming leave-one-out cross validation for k-means to complement this approach ...

(it is possible to do this in a streaming style, see LWPR)

Re:Introducing Streaming K-Means in Spark MLlib 1.2 by michaelmior on Wednesday January 28, 2015 @03:55PM (#8962495)
I understand that's a drawback of k-means. I was just wondering if this was something that Spark solved natively. Looks like the answer is no for the time being. Thanks for the pointer to LWPR :)
Re:Introducing Streaming K-Means in Spark MLlib 1.2 by rxin on Wednesday January 28, 2015 @03:58PM (#8962508)
Maybe using BP-Means: http://arxiv.org/abs/1212.2126

K-Means without preset K.

Re:Introducing Streaming K-Means in Spark MLlib 1.2 by cfregly on Wednesday January 28, 2015 @02:10PM (#8961832)
very interesting post. ironically, hackernews uses a similar type of time-decay algorithm!