Re:Building a scalable time-series database on PostgreSQL
by jjirsa on Wednesday April 05, 2017 @02:25AM
(#14040130)
To be clear - Cassandra is great for time series but it's not a "time series database" - it just solves all of the same problems (and then some), because it gives you natural replication, real HA, cross-wan, let's you cluster/order data on disk using big table like sorting, and scales over a thousand nodes per cluster.
For sensor data collection, it's very hard to beat. Lay something like spark or filodb on top for more flexibility and it's virtually unmatched.
I work a step upward in the stack (analyzing the sensor data and building models based off of it - usually using SQL) so I'm not an expert on underlying technology.
Over the last 15 years or so my work (industrial plant) has been big on historians. We've moved from in house bespoke stuff from the 90's which as I understood it was basically a custom written ring buffer that held 1-2 days of live data and would get down sampled into oracle tables as 1min/ 6min/hourly averaged data every hour or so.
Prior to this in the 80's we used a database technology called "IMS" but I'm not as familiar with how this system worked.
Nowadays in all our newer systems all I hear from the low level guys is "OPC" (which I gather is a messaging format) and historians. Time series databases sound like they should fit in somewhere but I'm not sure where - given other people use them for logging high frequency sensor data sounds like they could be a solution.