Re:How PostgreSQL aggregation works and how it inspired our ...
by djk447 on Thursday August 05, 2021 @08:44PM
(#28081955)
Absolutely! We're actually developing a lot of that: https://github.com/timescale/timescaledb-toolkit/tree/main/d...
A number of the things you're looking for we've done experimentally and we'll be stabilizing over the next few releases. So we'd love some feedback while we're still able to futz with the API without making breaking changes.
But the two you're asking about are, I think, going to be covered by hyperloglog (we just reimplemented the internals with HLL++) and stats_agg family of functions, which have both 1D (which will give you avg, stddev, variance, etc) and 2D (co-variance, slope, intercept, x-intercept etc as well as all the 1D functions).
Would also love issues if you think we're missing other stuff, going to be generalizing this and want to make it useful for folks.
(NB: Post author here.)
This is really useful for this kind of metric (distinct count) that can't be trivially aggregated. It allows generating pre-aggregated tables containing the intermediate aggregation state. Now you can compute your DAUs, WAUs and MAUs from the same daily pre-aggregates.
I wish this design was generalized in SQL. For instance, getting the same family of function for variance computation would be super useful for efficient & correct computation of confidence intervals at different levels of aggregation.