I work a step upward in the stack (analyzing the sensor data and building models based off of it - usually using SQL) so I'm not an expert on underlying technology.
Over the last 15 years or so my work (industrial plant) has been big on historians. We've moved from in house bespoke stuff from the 90's which as I understood it was basically a custom written ring buffer that held 1-2 days of live data and would get down sampled into oracle tables as 1min/ 6min/hourly averaged data every hour or so.
Prior to this in the 80's we used a database technology called "IMS" but I'm not as familiar with how this system worked.
Nowadays in all our newer systems all I hear from the low level guys is "OPC" (which I gather is a messaging format) and historians. Time series databases sound like they should fit in somewhere but I'm not sure where - given other people use them for logging high frequency sensor data sounds like they could be a solution.
For sensor data collection, it's very hard to beat. Lay something like spark or filodb on top for more flexibility and it's virtually unmatched.
Why didn't you use Scylla, which is 3x to 10x faster than Cassandra?
We ran hundreds of machines (per cluster), and multiple clusters.
We don't use Scylla for dozens of reasons, including (but not limited to):
- The product was written before Scylla existed
- Our benchmarks didn't show Scylla as being 3x faster, let alone 10x faster
- Our machine count wasn't based on Cassandra limits, it was based on the amount of data we were comfortable storing on a given machine
- We don't throw shiny new tech into a critical stack just because someone says it's faster. I'd rather run code that's tested at scale. ONE of my Cassandra clusters there would have been 10x larger than any known Scylla deployment. At the same time, that deployment is at least 2 orders of magnitude smaller than other published Cassandra deployments (meaning largest known Cassandra deployments are 3 orders of magnitude larger than the largest known Scylla deployment)
The one thing I agree with the author: databases aren't where you want to have hipster tech. You want boring things that work. For me, Cassandra is the boring thing that works.
If there's a downside here, it's that this database is still a little too immature to have a great use-case.
But by all means, it's not my money being burned, so I'm not losing sleep over it. I'll just sit here and giggle as people talk about scaling Postgres and in my mind all I hear is single point of failure masters and vacuum to prevent wraparound in high write workflows
Does it waste a certain amount of effort in duplicated work and products? Sure. But we also get new projects that may try new approaches, gain different experiences, and train new developers which all helps the market expand. Ultimately it's better to have yet another version than nothing at all.
Also, regarding vacuuming, TimescaleDB does not suffer from it much under the normal workloads it targets because time-series data is largely immutable and deletions typically happen at the chunk level (i.e., dropping complete tables), avoiding any vacuuming during deletes and making wraparound vacuuming more efficient.
Wraparound vacuuming still sucks for high write workloads. I've been there. I've fought that problem in a high-write-throughput-no-delete-immutable-workload. I've seen it in person. You're still writing a lot of frozen txids to disk. Your slaves are still going to get the WAL command in a single-threaded WAL sender and fall behind in replication as that vacuum runs. You're still going to have pain trying to create an HA setup.