grammr

Performance depends heavily on the complexity of your continuous queries, which is why we don't really publish benchmarks. PipelineDB is different from more traditional systems in that not all writes are all created equal, given that continuous queries are applied to them as they're received. This makes generic benchmarking less useful, so we always encourage users to roughly benchmark their workloads to really understand performance.

That being said, millions of events per second should absolutely be doable, especially if your continuous queries are relatively straightforward as you've suggested. If the output of your continuous queries fits in memory, then it's extremely likely you'd be able to achieve the throughput you need relatively easily.

Many of our users use our Kafka connector [0] to consume messages into PipelineDB, although given that you're using protobufs I'm guessing your messages require a bit more processing/unpacking to get them into a format that can be written to PipelineDB (basically something you can INSERT or COPY into a stream). In that case what most users do is write a consumer that simply transforms messages into INSERT or COPY statements. These writes can be parallelized heavily and are primarily limited by CPU capacity.

Please feel free to reach out to me (I'm Derek) if you'd like to discuss your workload and use case further, or set up a proof-of-concept--we're always happy to help!

[0] https://github.com/pipelinedb/pipeline_kafka

How aggregations are performed are determined entirely by your own continuous view definitions [0]. In this case I'm guessing you'd want to include a time-based column in the aggregation GROUP BY clause.

And since PipelineDB is a PostgreSQL extension, you can use the timestamptz type (which includes timezone support), and in general you could pretty easily simply normalize your event timezones in your continuous view definitions. When you're reading aggregate data back out, you could cast the time-based column using whatever timezone the client prefers.

Thanks for the question--I hope that was helpful!

[0] http://docs.pipelinedb.com/continuous-views.html

PipelineDB co-founder here--I think this is a pretty fair take! I would also like to point out that the aggregate data stored in PipelineDB can still be further aggregated, processed, JOINed on etc. on demand as well.

Since a continuous view's output is simply stored as a regular table, you are free to run arbitrary SELECT queries on it to further distill and filter your results. PipelineDB's special combine [0] aggregate allows you to combine aggregate values with no loss of information for this very purpose.

The most common pattern among our user base is to aggregate time-series data into continuous views at some base level of granularity (e.g. by minute) and then aggregate over that for final results (e.g. aggregate down to hour-level rows for the date range my frontend has selected).

[0] http://docs.pipelinedb.com/aggregates.html#combine

Sure! So stream-stream JOINs actually haven't been requested by users as much as you'd think. Users have generally been able to get what they need by using topologies of transforms [0], output streams, and stream-table JOINs. Continuous queries can be chained together into arbitrary DAGs of computation, which turns out to be a very powerful concept when mapping out a path from raw input events to the desired output for your use case.

The primary issue in implementing stream-stream JOINs is that we'd essentially need to preemptively store every single raw event that could be matched on at some point in the future. Conceptually this is straightforward, but on a technical level we just haven't seen the demand to optimize for it.

That being said, you could just use a regular table as one of the "streams" you wanted to JOIN on and then use an stream-table JOIN. As long as the table side of the JOIN is indexed on the JOIN condition, an STJ would probably be performant enough for a lot of use cases. With PostgreSQL's increasingly excellent partitioning support this is becoming especially practical.

I also suspect that this is an area where integration with TimescaleDB could be really interesting!

[0] http://docs.pipelinedb.com/continuous-transforms.html

I'm Derek, one of the co-founders--excellent question!

The former. PipelineDB performs aggregations in memory on microbatches of events, and only merges the aggregate output of each microbatch with what's on disk. This is really the core idea behind why PipelineDB is so performant for continuous time-series aggregation. Microbatch size is configurable: http://docs.pipelinedb.com/conf.html.

This is precisely why PipelineDB has rich support for data structures such as HyperLogLog [0]. HLL's allow you to track distincts information using fixed-size HLLs that only grow to about 14KB while encoding uniques counts for billions of distinct values. The tradeoff is about a ~0.8% margin of error, which users generally find acceptable.

Furthermore, PipelineDB has a special combine [1] aggregate that allows you to combine data structures such as HLL across multiple rows with no loss of information. A simpler example would be average: to get the actual average of multiple averages you obviously can't simply take the average of all the averages. Their weights must be taken into account, and combine handles that.

The capability to combine aggregate values in this way generalizes to all aggregates in PipelineDB.

[0] http://docs.pipelinedb.com/aggregates.html#hyperloglog-aggre...

[1] http://docs.pipelinedb.com/aggregates.html#combine

The most impactful thing you can do here is ask the RDS team for this. If enough users ask them for it they'll eventually begin seriously considering it :)

I'm Derek, one of the co-founders--great questions!

> So you create a table, insert into it, and it's always empty. Is that right?

That is correct. Streams can only be read by continuous queries (e.g. you can't even run a SELECT on them).

> Does this work for any table in pg? How does pg know that the insert should NOT actually insert a row?

PipelineDB streams are represented as a specific kind of PostgreSQL foreign table [0], so only foreign tables created in a specific way will be considered streams. You can use triggers to write table rows and updates out to streams if you want to though.

[0] https://www.postgresql.org/docs/current/static/sql-createfor...

I'm Derek, one of the co-founders--that's an interesting way to frame it, I think that makes a lot of sense at a high level.

We're in contact with the TSDB founders (awesome and super smart guys!) and are in the early stages of figuring out an integration that makes sense. That's most likely going to happen.

To anyone interested: we'd love to hear and consider your ideas re: TSDB integration. Feel free to open an issue in either repo (or add to an existing one) and tell us more!

I'm Derek, one of the co-founders--thank you!

We're super happy with where Stride is at! We've continued to onboard customers in a few AWS regions and the infrastructure is rock solid at this point. Most users are ingesting 10k+ events/s and their analytics frontends are retrieving results in well under 100ms. We've gotten it to the point where it "just works" which has made Stride users' lives a lot easier at that scale.

And since the hard parts of Stride are powered by PipelineDB, an added benefit for us is that we now get a ton of super detailed instrumentation data about PipelineDB performance and behavior, which has helped make the open-source product quite a bit better.

We'll be moving Stride into self-service/GA next year--stay tuned!

We haven't looked too far into integrations with any existing systems at this point, but if there was significant user demand for it on both ends we'd definitely be open to it.

One thing I will mention here is that we do have plans to add support for persistent streams [0] after version 1.0.0 is released. We've learned a lot over the years about how our users/customers interact with streams in production and persistent streams will be built atop that foundation of understanding.

Please feel free to comment on that issue with your use case, requirements, etc. and we'll see what we can do!

[0] https://github.com/pipelinedb/pipelinedb/issues/1463

Hi there! I'm one of the PipelineDB founders. This description is correct. The unique thing about PipelineDB is that it doesn't store granular data. Once all aggregates are incrementally updated, the raw input rows as discarded and only aggregate output is stored.

This approach dramatically limits disk IO and long-term storage requirements, and enables super high performance in most cases on modest hardware.

PipelineDB has been used in production for nearly four years now and is used by Fortune 100 companies.

If you're open to sharing, I'd love to hear more about your potential use case. Please email me!

Hello! I'm Derek, one of the PipelineDB co-founders. The way using PipelineDB feels to users has always been a principal consideration to us and how we make design decisions, so a psychological benefit isn't a second-class citizen in our minds. With so many different tools to choose from nowadays, any friction at all (technical or non-technical) can be a showstopper. We've always strived to make PipelineDB as easy to use as possible, and the extension refactor is the grand finale of that continual effort as we approach 1.0.

Thank you for your input, and we hope you you'll find great success with PipelineDB in the future!

> Pipelinedb is annoying though in that it is a fork and not an extension.

Hi, I'm one of the PipelineDB co-founders--thanks for using our product! Making PipelineDB an extension is the most consistent piece of feedback we've received from our users, and I promise we're listening: PipelineDB 1.0 will be a standard PostgreSQL extension, incrementally rolled out via versions 0.9.8, 0.9.9, and 1.0.0.

Stride is a realtime analytics API built for scale and performance. It allows you to construct networks of continuous processing nodes that either materialize results (think high-throughput aggregation) or fire webhooks ("when this aggregate exceeds this value, POST to a url"). Each node is represented by a continuous SQL query, and nodes can be chained together by "tailing" other nodes.

These processing nodes can be queried and joined on (with SQL) at any time to easily power realtime dashboards and other analytical applications. Check out the original announcement [0] and the Stride docs [1] for more detail.

And to answer your question, no, Stride is not simply a frontend for PipelineDB. While it does use PipelineDB and PipelineDB Cluster extensively, it also uses other systems to provide unique capabilities, and all of this is ultimately synergized behind a dead-simple HTTP API.

We've found that a hosted database isn't actually all that interesting for users nowadays, and near impossible to build a business around because they've essentially become a cheap commodity. So we aim to deliver maximum value to users by nailing one use case (realtime analytics) with a highly focused API that eliminates most of the complexity and decision points you'd encounter when trying to do the same with a generic database.

[0] https://www.pipelinedb.com/blog/announcing-stride-a-realtime...

[1] https://www.stride.io/docs

(Hi! I'm one of the founders of PipelineDB)

PipelineDB will be a standard PostgreSQL extension[0] by release 1.0.0. Currently we're about to release 0.9.7, and each release after that will incrementally factor out PipelineDB into a completely independent PostgreSQL extension.

That's the plan, anyways :) With https://www.stride.io being rolled out under heavy demand, we've got a lot on our plate!

[0] https://github.com/pipelinedb/pipelinedb/issues/1596

Author here. The post is pretty high level so I'm more than happy to answer any questions or elaborate on anything here.

There was a bug in the example. Fixed, thanks!

We just updated http://www.pipelinedb.com/download/0.7.7/osx.

Please shoot me an email (I'm Derek) if you have any issues installing this package. Thanks for your patience!

Sorry about that, OSX packaging is a bit finicky right now. We'll publish a package with more sensible defaults very shortly.

Hey Chad, PipelineDB co-founder here. PipelineDB certainly isn't intended to be the only tool in your data infrastructure. But whenever the same queries are being repeatedly run on granular data, those are the types of situations in which it often makes a lot sense to just compute the condensed result incrementally with a continuous view, because that's the only lens it's ever viewed through anyways (dashboards are a great example of this). Continuous views can be further aggregated and queried like regular tables too.

In terms of not requiring that raw data be stored, a typical setup is to keep raw data somewhere cheap (like S3) so that it's there when you need it. But granular data is often overwhelmingly cold and never looked at again so it may not always be necessary to store it all in an interactively queryable datastore.

As I mentioned, PipelineDB certainly doesn't aim to be a monolithic replacement for all adjacent data processing technologies, but there are areas where it can definitely introduce significant efficiency.

Founder @ PipelineDB here (hi Slava!)

Currently continuous views must read from a stream. However, in the very near future it will possible to write to streams from triggers, which would probably give you enough flexibility to model the behavior you want if you could conceptualize a table as a stream of changes.

PipelineDB (YC W14) | Systems Engineers and Frontend Engineers | San Francisco | Full time | [email protected]

We're on a mission to build a new type of database for a modern world in which information is constantly moving, and moving fast. PipelineDB runs SQL queries continuously on large volumes of streaming data, giving companies the capability to easily develop scalable, realtime applications and services using only a familiar SQL interface. No application code is required.

This inherently involves solving a lot of big problems, many of which are novel. We’re looking for creative engineers who appreciate the value and freedom of choosing their own projects, approaches, and working with other top talent in a low distraction, streamlined work environment. Our small team has backgrounds from Berkeley, MIT, Facebook, Locu and AdRoll, and we're all doing exactly what we want to be doing: building a groundbreaking new product out of thin air.

As an early stage engineer you'll ultimately own a very large part of the product. Which part of the product you take charge of depends on where your interests are, but there are several different potential areas of focus. You'll be entrusted to make sound architectural decisions as well as implement your vision effectively.

We are well funded by top investors including SV Angel, Susa Ventures, Data Collective, Paul Buchheit, and more.

If you’ve been waiting for an opportunity like this, please send your resume and a quick blurb about yourself to [email protected].

Benefits:

* Full medical/dental/vision insurance

* No set work hours--work when you feel smart

* Choose your own setup

* No vacation policy other than that it is strongly encouraged

* Large equity ownership

PipelineDB | Systems Engineer | San Francisco | Full time | [email protected]

We're on a mission to build a new type of database for a modern world in which information is constantly moving, and moving fast. PipelineDB runs SQL queries continuously on large volumes of streaming data, giving companies the capability to easily develop scalable, realtime applications and services using only a familiar SQL interface. No application code is required.

This inherently involves solving a lot of big problems, many of which are novel. We’re looking for creative engineers who appreciate the value and freedom of choosing their own projects, approaches, and working with other top talent in a low distraction, streamlined work environment. Our small team has backgrounds from Berkeley, MIT, Facebook, Locu and AdRoll, and we're all doing exactly what we want to be doing: building a groundbreaking new product out of thin air.

We are well funded by top investors including SV Angel, Susa Ventures, Data Collective, Paul Buchheit, and more.

If you’ve been waiting for an opportunity like this, please send your resume and a quick blurb about yourself to [email protected].

About the systems engineering position:

As an early stage engineer you'll ultimately own a very large part of the product. Which part of the product you take charge of depends on where your interests are, but there are several different potential areas of focus. You'll be entrusted to make sound architectural decisions as well as implement your vision effectively. We review each others' code for quality, awareness, and learning, so you'll spend some time contributing your valuable perspective to those efforts as well, although the vast majority of your time will be spent designing and building your own projects.

Benefits:

* Full medical/dental/vision insurance

* No set work hours--work when you feel smart

* Choose your own setup

* No vacation policy other than that it is strongly encouraged

* Large equity ownership

PipelineDB (YC W14)

We are on a mission to build a new type of database for a modern world in which information is constantly moving, and moving fast. PipelineDB runs SQL queries continuously on large volumes of streaming data, giving companies the capability to easily develop scalable, realtime applications and services using only a familiar SQL interface. No application code is required.

This inherently involves solving a lot of big problems, many of which are novel. We’re looking for creative engineers who appreciate the value and freedom of choosing their own projects, approaches, and working with other top talent in a low distraction, streamlined work environment in our new SOMA office.

We are well funded by top investors including SV Angel, Susa Ventures, Data Collective, Paul Buchheit, and more.

If you’ve been waiting for an opportunity like this, please send your resume and a quick blurb about yourself to [email protected]. We're hiring for two positions:

1) Software Engineer - San Francisco, CA

Requirements:

* Bachelor's degree or higher in CS or related field

* You've worked with large C/C++ systems in a Linux environment

* Strong system-level debugging skills

* Strong understanding of how performant storage systems work

Bonus points:

* Experience with/contributions to PostgreSQL, Storm, Kafka, Samza, Kinesis, or Esper

* Experience with database internals

* Experience building distributed systems

* Experience building realtime systems

Compensation:

$90k - $120k + 2% - 4% equity

--

2) Frontend Engineer - San Francisco, CA

Requirements:

* Strong design sense (please send us a link to your portfolio/work if possible)

* Experience with a framework such as Django or Rails

* Strong HTML, CSS, and JavaScript skills

* Ability to implement your vision effectively

Bonus points:

* Bachelor's degree or higher in CS or related field

* Experience with databases

* Experience with monitoring/operations/admin software

Compensation:

$80k - $110k + 1% - 3% equity

--

[email protected]