News for hackers. Stuff that matters.
Re:Timescale, an open-source time-series SQL database for Po... by mfreed on Friday August 11, 2017 @08:20AM (#14988873)
Well...sorta. Postgres/Timescale have pretty rich support for JSON these days (and its more efficient "binary" JSONB), so there are a whole range of options you can do that make it feel much more schema-less than before.

In fact, last month we released a beta version of a Prometheus connector for Timescale/Postgres that allows you to store arbitrary Prometheus metrics without pre-defining all these varied schemas:

https://github.com/timescale/pg_prometheus

The same approach should work for Telegraf, we just haven't yet tried to generalize this plugin.

Re:Timescale, an open-source time-series SQL database for Po... by koffiezet on Friday August 11, 2017 @09:00AM (#14989192)
Well, JSONB has a performance or afaik a pretty significant disk usage impact if you use GIN. It's pretty nice stuff, but I'm not sure about using it for timeseries. I haven't seen any benchmarks though, this could be useful.

One advantage specialized db's like influx have is specialized/optimized storage layers for the type of data while timescale seems to use normal postgres tables behind the scenes.

That Prometheus extension and adapter however look nice! They seem to be a good drop-in replacement for whatever storage, and the missing link for anything that's able to talk to Prometheus (which is quite a lot, including Telegraf).

Re:Timescale, an open-source time-series SQL database for Po... by mfreed on Friday August 11, 2017 @09:22AM (#14989353)
A common approach we find for storing devops-related time-series data (as you'd find with influx), is to not denormalize the "labels" or "tag set" into the main metrics table.

This obviously saves significant space just by avoiding denormalization, ignoring the indexing overhead as well. You can see that in our Prometheus extension, btw:

https://github.com/timescale/pg_prometheus#user-content-norm...

Regarding performance against Influx, it really depends. We're working on releasing more complex benchmarks soon.

But overall, they have a performance edge if it's a single column scan that precisely matches their particular architecture (e.g., WHERE clause or GROUP BY by a very specific label set). But, we've found that Timescale actually gets higher query performance across a whole set of pretty natural queries (sometimes ridiculously so, as Influx doesn't index numerical values). Plus higher insert rates and full SQL. Not only is the latter point important for enabling many more types of complex queries, but it means that any viz or reporting tool that already speaks to Postgres should just work with us.

Re:Timescale, an open-source time-series SQL database for Po... by koffiezet on Monday August 14, 2017 @05:09AM (#15007404)
> But overall, they have a performance edge if it's a single column scan that precisely matches their particular architecture

To be honest if they didn't - stuff like Influx wouldn't really have a reason to exist. I think 99.9% of the operations in a devops environment will be pretty simple and predictable, it's a a trade-off they make. Queries are mostly there to draw graphs quickly.

I quite like the simplicity of Influx, and while our current use is pretty limited and far from hitting it's limits, for reliability and maintenance reasons I'd prefer Postgres, for which we have quite extensive in-house know-how and tools in place. Prometheus is one of the tools we are considering to add to our monitoring stack, and having the option of storing it's data transparently in Postgres could be very interesting.