Re:Ask HN: Best time series database currently
by johann8384 on Thursday November 30, 2017 @05:43PM
(#15820325)
If you were using OpenTSDB: 1 million devices, reporting once per minute. We'll round that up to 17k submissions per second. Let's say each submission is, um, 50 datapoints. That's 850,000 datapoints per second. Not really that hard to store. With 12 tags, OpenTSDB stores that in about 100 bytes per datapoint. You'd be writing 80MBps of data across you cluster. Not a big deal there. Let's say you have 12 nodes with 8 disks at 2TB per disk. 192TB with 3x HDFS replication, so 64TB of usable space. That gives you 828,504 minutes of capacity. 19 months or so. Queries will be fast also, but you need to set it up well, and, it's easy to do it wrong (sorry about that, we're making it better). I recommend Splicer with 4 or 5 query instances per data node to parallelize the queries and take advantage of locality (hit the query instances on the same node as the HBase region).
I missed where the Petabyte of data came in, but I just made up some numbers here that are accurate from my experience.
Re:Ask HN: Best time series database currently
by mads on Thursday November 30, 2017 @05:51PM
(#15820391)
I recently built a system capable of handling this load, but is was based on MongoDB and thats not an option in this project because of hardware constraints.
Cassandra may be an option. I will look into it.
To know how to store the data, you need to know how you will query it.
If you know how you will query it, you can say devise a way to store it in Cassandra.. which will scale up to PB.
If you just throw in a PB of data, I am not aware of ANY system that is going to let you drop ad-hoc queries at the data and get fast answers. You effectively need to load most / much of the data off disk to process it.
If you only want to store it and not query it yet.. store it off on flatfile. When you decide how you will query it, load it from the flatfile and switch writes to your new system.