You know, that sounds like exactly the kind of thing you'd want to do in a DevOps world (leverage your data and new ideas to predict problems).
So, you're not just right that we need to look at timeseries because there are many interesting new problems that we might solve with them, but also because there are still many problems we want to solve in DevOps.
It's kind of now accessible to the masses (previously you'd have to pay SAP a few million for BusinessObjects then EMC another million and a half for enough storage to setup a DW), but open source tools have been available for at least a decade. Now it just has a fancy new marketable name "predictive analysis" that Oracle et al can charge an extra couple million for with their price-gouging RAC licenses.
I recall that Kibana is a plugin for Elastic Search but don't know about the others. Isn't it possible to connect a hypothetical standalone visualization frontend to any TSDB?
The TSDBs I tested when I was building this setup (Graphite, OpenTSDB and InfluxDB) all had some sort of basic plotting facilities, but they were extremely minimal and all pointed towards Grafana if you wanted something more substantial.
Have you documented your setup anywhere?
I'm DIY-ing something similar and I'm always curious how others have approached things.
We're also in the process now of finding a good storage solution for time series data at my dayjob. We're not storing any server metrics, but more personal user health data and related metrics. So we want the flexibility of being able to write new metrics on your personal timeline without altering a schema. Most reads would be querying single users timeline to fetch their data and deliver through APIs for rendering in clients. Also of course to do analysis on all users timelines, find correlations etc. but those queries are less frequent and not so time critical. I'm starting working on a prototype now with MongoDB. Other DBs that have come up has been InfluxDB, Riak TS, Amazon DynamoDB and probably some others I don't remember. Haven't actually thought that much about Postgres, but thanks for your links to your blog posts I will read up on how Postgres might work. Most other non-time series data would still be in our MySQL setup.
While it is used for machine learning, it actually makes a lot of sense to follow a similar approach for more general time series applications.
That's making some presumptions about the time-series database use case. SAX is convenient for storing and retrieving the data as well as identifying trends or recurring behaviour. What more do you need?
Yep. Each SAX word should be mapped to the list of seriesid:timestamp pairs. This list is often referred as postings list in information retrieval. The resulting data-structure is an inverted index. SAX and iSAX papers describes inverted index variant (really bad one) based on folder structure but one can use convenient IR tools for this.
The thing is, particularly with time series data, a lot of times it is sufficient to at least start with the summary data in the index.
My issues with the present state of TS isn't the volume. I was looking for using TS outside of the DevOps world. Everyday things like your heart rate over time, price of gas at the nearest station, number of people in line at your coffee shop, etc. All these are interesting, and Graphite/RRDTool/InfluxDB/etc did not seem like appropriate storage, because to use it with your other data (which is most likely in a relational DB of some kind) you need to export/import it and who wants that.
I call this problem "data seclusion". When data exists in some kind of an incompatible format (e.g. Whisper files), it will end up ignored because of that extra conversion step necessary to link it with your other data. Data in Graphite and such is mostly good for generating charts, but TS analysis is so much more than that, even at its simplest.
I think that the good old relational database is fine storage for TS and we gave up on it way too early, especially given what's new in PostgreSQL. Making it horizontally scalable, distributed, using consensus protocols, etc - these are not time series problems, these are database problems and we do not yet have a good solution for these. (We have many that "kind of" work, support some features but not others e.g. Cassandra). Projects like InfluxDB are mired in solving the wrong problem which will eventually get solved at the DB level.
More thoughts on the subject: http://grisha.org/blog/2015/03/28/on-time-series/ http://grisha.org/blog/2015/09/23/storing-time-series-in-pos... and http://grisha.org/blog/2015/05/04/recording-time-series/