Re:Show HN: Matrixprofile-ts – A Python library for timeseri...
by jamesb93 on Sunday January 06, 2019 @09:38PM
(#18842314)
You say that the algorithm is fast, and the literature certainly points to this too but I tried the python implementation (linked here) on some audio data sets. 1 second of audio at a reasonable quality is 44.1k data points and it was taking minutes to process this data.
I tried an R implementation which was multi-threaded and a lot faster, but still the algorithm took ages to test lots of different window sizes and data sets.
Re:Show HN: Matrixprofile-ts – A Python library for timeseri...
by gdpq11 on Monday January 07, 2019 @09:28PM
(#18852570)
That's odd, we've definitely processed larger datasets much quicker than that. Feel free to raise an issue on Github and we can take a look.
Re:Show HN: Matrixprofile-ts – A Python library for timeseri...
by jamesb93 on Wednesday January 09, 2019 @10:01AM
(#18865144)
Just to advance my previous comment; right now the most interesting feature to me is the motif detection, however, the motif's are always incredibly short (or a fixed size?). Please excuse me ignorance in this regard, I'm not too versed in the algorithm. Is there any use cases where you have looked at longer motifs?
Re:Show HN: Matrixprofile-ts – A Python library for timeseri...
by gdpq11 on Thursday January 10, 2019 @01:12PM
(#18876318)
There's some interesting work around multi-month seasonality that covers what you're talking about, but Eamonn Keogh would probably have a better answer that I :). Also, in regards to your performance issue, did you use the pip version or the code directly from Github?
Re:Show HN: Matrixprofile-ts – A Python library for timeseri...
by jamesb93 on Wednesday January 16, 2019 @08:14AM
(#18920514)
I used the pip version.
Re:Show HN: Matrixprofile-ts – A Python library for timeseri...
by jamesb93 on Tuesday January 08, 2019 @05:14AM
(#18854556)
Cool. Thanks for responding. I'll provide some example code and my specific use case.
Re:Show HN: Matrixprofile-ts – A Python library for timeseri...
by eamonn on Thursday January 10, 2019 @01:55AM
(#18872041)
Does this help..
https://www.cs.ucr.edu/~eamonn/public/GPU_Matrix_profile_VLD...
Re:Show HN: Matrixprofile-ts – A Python library for timeseri...
by gdpq11 on Thursday January 10, 2019 @01:15PM
(#18876352)
Definitely! We see GPUs as the primary method of calculating the Matrix Profile given that Big Data is only getting bigger. So, advancing from STOMP-GPU to SCAMP will be very helpful.
In that case, you'd create a parameter "w" that specifies the boundary between when two matching points are a pattern, or if enough time has elapsed so that they should be considered two anomalies. In the NxN matrix, for the ith row you'd then set every value outside the i+w/i-w boundary to infinity. In that way, the resulting Matrix Profile would account for your situation.
Due to the algorithm's speed we do often sweep over multiple values, but try to use domain knowledge where we can. And for alerting, we sometimes have labeled data that we can calibrate the threshold to, but often times that's a matter of customer trial and error.