Re:KSQL: Open Source Streaming SQL for Apache Kafka
by Fergi on Monday August 28, 2017 @08:55PM
(#15121389)
PipelineDB will become a standard, open-source PostgreSQL extension this year, enabling anybody using PostgreSQL to leverage continuous SQL queries in combination with all other PostgreSQL functionality.
https://www.pipelinedb.com/blog/pipelinedb-0-9-8-postgresql-...
Re:KSQL: Open Source Streaming SQL for Apache Kafka
by _pmf_ on Tuesday August 29, 2017 @08:15AM
(#15123735)
> PipelineDB will become a standard, open-source PostgreSQL extension
What does this imply?
Re:KSQL: Open Source Streaming SQL for Apache Kafka
by cbsmith on Monday August 28, 2017 @09:16PM
(#15121465)
Are InfluxDB's continuous queries really like this? It's so much less of a streaming solution I hadn't realized it did this.
Re:KSQL: Open Source Streaming SQL for Apache Kafka
by stuartaxelowen on Monday August 28, 2017 @10:12PM
(#15121638)
Doesn't PipelinDB limit you to postgres' scaling capabilities?
Re:KSQL: Open Source Streaming SQL for Apache Kafka
by Fergi on Monday August 28, 2017 @10:28PM
(#15121686)
Not for streaming analytic workloads, because PipelineDB fundamentally adds continuous queries to PostgreSQL, so data is continuously distilled and aggregated as it arrives, before it is stored, which drastically reduces the amount of data stored in PipelineDB (or soon, in PostgreSQL via the extension refactor).
PipelineDB also offers a clustering extension for large workloads (see: http://enterprise.pipelinedb.com/docs/)
But in terms of ad hoc, exploratory analytics workloads, yes - the scaling limitations would be the same, since for ad hoc, exploratory analytics PipelineDB and PostgreSQL are the same. But with that said, the processed, aggregated data that gets stored is generally much smaller than large volumes of granular data, so there is much less data to comb through with PipelineDB.
Re:KSQL: Open Source Streaming SQL for Apache Kafka
by stuartaxelowen on Tuesday August 29, 2017 @12:28AM
(#15122077)
Thanks for explaining!
Re:KSQL: Open Source Streaming SQL for Apache Kafka
by sandGorgon on Tuesday August 29, 2017 @06:42AM
(#15123368)
this is awesome! will it work on Amazon RDS for postgresql ?
influxdb has something similar, and now so does AWS Kinesis (with their Kinesis Analytics product).