dkgs

Hi!

Will definitely add those to the docs and landing page — really appreciate the feedback! :)

We originally built Pivot for our internal observability pipeline, where we streamed the same data into both ClickHouse (with 1 month TTL - for fast dashboards and customer-facing analytics on recent data) and Iceberg (for longer data archiving and ETL jobs). We were frustrated with this duplication (and with ClickHouse OSS being disk-based rather than S3-based!), and got curious about how fast an Iceberg-native real-time analytics database could be!

Re DuckDB — it's definitely an amazing piece of technology, but it didn't provide the full "database" experience we needed: ingestion, efficient compaction, high concurrency, cluster deployment option, and a single endpoint for our backends and dashboards to query.

Ultimately, we managed to cut the cost of our customer-facing analytics and observability pipeline by around 75%, so we figured it might be useful to others as well! :)

Re hardware variation, we definitely optimized for cloud native deployments.

While we do have some optimizations and kernels for specific instruction sets on both x86 and ARM (for example AVX-512 on x86, and with Armv8.2-A as the minimum for ARM), most of our optimizations focus on taking advantage of the memory and network bandwidth of modern hardware, both of which have increased significantly in recent years (https://pivotlake.io/docs/database/why-pivot-is-fast/#utiliz...).

Hi HN!

This is Gilad, one of the creators of Pivot

We built an open-source analytics database that allows companies to run real-time analytics workloads on top of open data formats (Iceberg, Parquet), instead of needing to store data in a closed-format datastore (e.g., ClickHouse, Druid…) for real-time analytics use cases.

We've built the engine from scratch in Rust and use Arrow for in-memory data processing, alongside a thread-per-core execution model to fully control the scheduling of CPU work. To achieve the speed we wanted, we had to build a custom memory allocator, tinker with the way JOIN and GROUP BY algorithms work to fully utilize the memory bandwidth of modern CPUs, and, in general, do a lot of optimization work to ensure we achieve superior performance compared to other real-time analytics engines, while remaining committed to using only Parquet / open data formats!

We're excited to share it with the community and hope others find it as useful as we have!

Hi HN! One of the creators of Pivot here :)

Happy to answer any questions about Pivot, its architecture, or the decisions we made building it!

A coincidence with the fact duckdb is supposed to release 2.0.0 very soon? :)

Acknowledging the benifits of keeping a commercial product closed source is one thing, claiming open source has no value is another thing.

Sad to see this coming from a company that was itself built on the foundation of an open-source project like Vitess.

Nothing, why would that be an issue (IE the same client submitting requests with two identical UUIDs)?