News for hackers. Stuff that matters.
Re:Building a distributed time-series database on PostgreSQL by theomega on Wednesday August 21, 2019 @05:32PM (#20762063)
I can only recommend TimescaleDB. It solves the right problems (storing timeseries) while not creating new ones (deployment, backup, hot failover) as it relies on Postgres to provide the underlying infrastructure. I stored 100 million sensor samples in TimeScale and had not issues with scaling on medium sized boxes, despite issuing complex time-series queries.

As for the hosting option, currently sadly AWS doesn’t offer Timescale as part of RDS. There are two options: Azure offers Timescale now as part of their hosted Postgres. Or you go with aiven.io who can host you postgres with TimeScaleDB on all cloud providers (AWS, GCP, Azure, DO, ?) as a service, including replicas and backups.

Overall, I’m very happy to see the Postgres ecosystem growing.

Re:Building a distributed time-series database on PostgreSQL by akulkarni on Wednesday August 21, 2019 @06:00PM (#20762311)
If you are looking for hosted TimescaleDB, eg on AWS, you may want to check out Timescale Cloud [1], which we launched a couple months ago.

Fully-managed TimescaleDB, including community and enterprise capabilities, high-availability, etc, available on AWS, GCP, and Azure.

There are other options as well (Azure, DigitalOcean, Alibaba, Aiven) but they only offer the OSS version of TimescaleDB.

More here (including a pricing calculator): https://www.timescale.com/cloud

[1] https://blog.timescale.com/blog/timescale-cloud-first-fully-...

Re:Building a distributed time-series database on PostgreSQL by stalller on Wednesday August 21, 2019 @06:03PM (#20762340)
Thanks for the recommendation! And glad to hear TimescaleDB meets your needs. I wanted to add some color to hosting / deployment options around TimescaleDB.

Options are as follows:

[1] Fully-Managed TimescaleDB Enterprise on Timescale Cloud

TimescaleDB Open Source hosted on Public Clouds:

[2] Azure PostgreSQL

[3] DO Managed-Postgres (mentioned above)

[4] Alibaba Cloud

Not hosted, but managed service options:

[5] ClusterControl from our friends at Severalnines

---

[1]https://www.timescale.com/cloud [2]https://azure.microsoft.com/en-us/blog/power-iot-and-time-se... [3]https://www.digitalocean.com/docs/databases/postgresql/resou... [4]https://www.alibabacloud.com/blog/sql-and-timescaledb_595169 [5]https://severalnines.com/blog/advanced-database-monitoring-m...

Re:Building a distributed time-series database on PostgreSQL by brootstrap on Wednesday August 21, 2019 @05:40PM (#20762133)
Interesting. My team currently uses (abuses?) postgres for timeseries data. You mind ansswering some general questions about your experience with timescale? You said 100 million sensor samples. What was the upload/download frequency? Our application is pushing hundreds of millions of rows across many different data sources every day. On top of that, we are also querying the shit out of this data to run models and we need VERY quick queries. like 10-100ms speed.

How do you think timescaleDB would handle that size and also velocity of data?

Re:Building a distributed time-series database on PostgreSQL by valyala on Thursday August 22, 2019 @12:53PM (#20770073)
TimescaleDB could fit your workload if PostgreSQL fits you. The main issue with PostgreSQL and TimescaleDB is big amounts of storage space required for huge time series data volumes. There are reports that storing data on ZSF can reduce the required storage space.

Probably, ClickHouse [1] would fit better your needs. It can write millions of rows per second [2]. It can scan billions of rows per second on a single node and it scales to multiple nodes.

Also I'd recommend taking a look at other open-source TSDBs with cluster support:

- M3DB [3]

- Cortex [4]

- VictoriaMetrics [5]

These TSDBs speak PromQL instead of SQL. PromQL is specially optimized query language for typical time series queries [6].

[1] https://clickhouse.yandex

[2] https://blog.cloudflare.com/http-analytics-for-6m-requests-p...

[3] https://www.m3db.io/

[4] https://github.com/cortexproject/cortex

[5] https://github.com/VictoriaMetrics/VictoriaMetrics/

[6] https://medium.com/@valyala/promql-tutorial-for-beginners-9a...

Re:Building a distributed time-series database on PostgreSQL by akulkarni on Thursday August 22, 2019 @05:13PM (#20772404)
We are actively working on native compression in TimescaleDB, with really promising early results. More to come.
Re:Building a distributed time-series database on PostgreSQL by merb on Thursday August 22, 2019 @04:02AM (#20765727)
actually the last time I checked/used it timescaledb did use parallel queries and had a good partinoning so that the parallel queries would even scale really good on a single node.
Re:Building a distributed time-series database on PostgreSQL by laughinghan on Wednesday August 21, 2019 @05:43PM (#20762157)
DigitalOcean Managed Databases also comes with TimescaleDB built-in: https://www.digitalocean.com/docs/databases/postgresql/resou...
Re:Building a distributed time-series database on PostgreSQL by GordonS on Wednesday August 21, 2019 @06:02PM (#20762331)
I was going to post a very similar comment :)

For on-prem deployments I'm using TimescaleDB on a single node with up to 100 million events, and for SaaS I'm using it on Azure, and the performance is little short of amazing!

Something I particularly like is the almost instantaneous deletes (because of the "chunking" model) - perfect for data retention jobs that delete old data.

Re:Building a distributed time-series database on PostgreSQL by keithnoizu on Thursday August 22, 2019 @03:46AM (#20765631)
I use RiakTS although unfortunately with Basho's demise it's future is in a bit of jeopardy.