News for hackers. Stuff that matters.
Re:How Time Series Databases Work, and Where They Don’t by digbybk on Monday October 18, 2021 @09:48AM (#28905406)
Can you talk more about why you should use Postgres unless you have a compelling reason not to? I'm currently investigating options for storing high-frequency data. Postgres is looking like a good option.
Re:How Time Series Databases Work, and Where They Don’t by gilbetron on Monday October 18, 2021 @10:51AM (#28906014)
Postgres has a lot of engineering work put into it to handle all kinds of use cases, plus massive effort by the community building tutorials, videos, wrappers, libraries, etc. If you have a problem with Postgres config, or are trying to get it to do something odd, there is undoubtedly a bunch of Stack Overflow discussions about that thing. For most other databases, the selection of all of those is much thinner.

Additionally, often we believe our application needs feature X, and there is some database tech that purports to excel at X, the fact is postgres is probably able to do X unless you get to some extreme velocity or volume. Furthermore, by going with another database you are almost always giving up on features W, Y, and Z that you don't realize you need that Postgres supports and isn't supported by the "Exotic" database you are thinking of.

In short, Postgres has amazing breadth and depth in features and support and tooling. Be sure you are ok giving that up!

Re:How Time Series Databases Work, and Where They Don’t by akulkarni on Monday October 18, 2021 @01:16PM (#28907943)
If you like Postgres, you may want to try TimescaleDB, which is a time-series database built on Postgres (packaged as a Postgres extension). Postgres database + time-series database all in one.

This btw is one of the reasons I love Postgres - its extensibility.

(Disclaimer: TimescaleDB cofounder)

Re:How Time Series Databases Work, and Where They Don’t by gilbetron on Monday October 18, 2021 @01:45PM (#28908292)
Did you guys used to call it just ScaleDB? I remember talking to some people about something called ScaleDB that was built on top of Postgres back when I was looking for database solutions for another product, this was in 2015. We ended up going with Druid for that.
Re:How Time Series Databases Work, and Where They Don’t by proddata on Monday October 18, 2021 @03:09PM (#28910086)
What do yo mean by high-frequency data? 100Hz, 1KHz, 100KHz? For that kind of use cases many time-series DBs break apart. We have customers storing multiple millions of high frequency measurements per sec in arrays.

I would say, Postgres is not too storage efficient in itself for large amounts of data, especially if you need any sorts of indexes. Timescale basically mitigates that by automatically creating new table in the background ("chunks") and keeping individual tables small.

Re:How Time Series Databases Work, and Where They Don’t by LoriP on Tuesday October 19, 2021 @06:00AM (#28916506)
TimescaleDB also implements compression. From the docs:

When compression is enabled, TimescaleDB converts data stored in many rows into an array. This means that instead of using lots of rows to store the data, it stores the same data in a single row. Because a single row takes up less disk space than many rows, it decreases the amount of disk space required, and can also speed up some queries.

(Timescale employee)

Re:How Time Series Databases Work, and Where They Don’t by digbybk on Monday October 18, 2021 @09:27PM (#28913889)
Generally in the 100Hz or 200Hz range for the time being. What do you mean by break apart?
Re:How Time Series Databases Work, and Where They Don’t by proddata on Wednesday October 20, 2021 @02:50AM (#28927574)
Not being able to keep up with the incoming data. But 100-200Hz I'd consider fine for most
Re:How Time Series Databases Work, and Where They Don’t by mehanik on Monday October 18, 2021 @05:52PM (#28912279)
We tried to use Postgres with TimescaleDB plugin for high frequency data several TB in size. It was unusable. Switched to Clickhouse, which was roughly 50-100 faster on the same hardware and 10 times less disk space. They use very different storage engines with different functionality so check the docs to see what fits your use case.