News for hackers. Stuff that matters.

A comment on Building a distributed time-series database on PostgreSQL. Parent comment

Re:Building a distributed time-series database on PostgreSQL by brootstrap on Wednesday August 21, 2019 @05:40PM (#20762133)
Interesting. My team currently uses (abuses?) postgres for timeseries data. You mind ansswering some general questions about your experience with timescale? You said 100 million sensor samples. What was the upload/download frequency? Our application is pushing hundreds of millions of rows across many different data sources every day. On top of that, we are also querying the shit out of this data to run models and we need VERY quick queries. like 10-100ms speed.

How do you think timescaleDB would handle that size and also velocity of data?

[ Parent | on HN ]
Re:Building a distributed time-series database on PostgreSQL by valyala on Thursday August 22, 2019 @12:53PM (#20770073)
TimescaleDB could fit your workload if PostgreSQL fits you. The main issue with PostgreSQL and TimescaleDB is big amounts of storage space required for huge time series data volumes. There are reports that storing data on ZSF can reduce the required storage space.

Probably, ClickHouse [1] would fit better your needs. It can write millions of rows per second [2]. It can scan billions of rows per second on a single node and it scales to multiple nodes.

Also I'd recommend taking a look at other open-source TSDBs with cluster support:

- M3DB [3]

- Cortex [4]

- VictoriaMetrics [5]

These TSDBs speak PromQL instead of SQL. PromQL is specially optimized query language for typical time series queries [6].

[1] https://clickhouse.yandex

[2] https://blog.cloudflare.com/http-analytics-for-6m-requests-p...

[3] https://www.m3db.io/

[4] https://github.com/cortexproject/cortex

[5] https://github.com/VictoriaMetrics/VictoriaMetrics/

[6] https://medium.com/@valyala/promql-tutorial-for-beginners-9a...

[ Parent | on HN ]
Re:Building a distributed time-series database on PostgreSQL by akulkarni on Thursday August 22, 2019 @05:13PM (#20772404)
We are actively working on native compression in TimescaleDB, with really promising early results. More to come.
[ Parent | on HN ]
Re:Building a distributed time-series database on PostgreSQL by merb on Thursday August 22, 2019 @04:02AM (#20765727)
actually the last time I checked/used it timescaledb did use parallel queries and had a good partinoning so that the parallel queries would even scale really good on a single node.
[ Parent | on HN ]