News for hackers. Stuff that matters.
Re:Fast drilldown dashboards from a single Parquet file by simonw on Monday August 24, 2026 @10:41AM (#49420491)
"static datasets with range payloads small enough to fit into a web response" fits a lot of workloads.

I expect that if your overall data is less than a GB this trick will work really well for you.

Re:Fast drilldown dashboards from a single Parquet file by nicoritschel on Monday August 24, 2026 @11:50AM (#49421528)
Most reporting I've ever worked on is based on live data and users expect updates. Range requests over parquet cubes are a cool party trick, but you outgrow it quickly once you need to start regularly updating the dataset such as to avoid full recompuation.

The next step in this journey is Iceberg (and a proper incremental pipeline), which can also be read directly in the browser via WASM either via DuckDB or without. This is from the same author as the parquet library mentioned in the OP https://github.com/hyparam/icebird

Re:Fast drilldown dashboards from a single Parquet file by hamilton on Monday August 24, 2026 @12:45PM (#49422464)
I think it depends on (1) customer expectations for freshness and (2) scale (both for the cubes and for the customer data in toto). There are many types of customer facing dashboards where giving "live data" is a bad idea for them and for you. And recomputation is indeed a problem, but if the volume of data isn't that high to begin with, it's probably easier than setting up an incremental pipeline architecture vs. a grouping set query in DuckDB. But I am not really a data engineer, so perhaps this is naive.
Re:Fast drilldown dashboards from a single Parquet file by closeparen on Monday August 24, 2026 @10:29PM (#49428402)
I work at a household name tech company and pretty much all our "data" dashboards are running SQL queries against a Hive/Presto environment that's at least 24 hours behind. Real-time dashboards are limited to the service metrics time series database (so combinations of categorical variables, no individual records) or rare expensive Pinot tables.
Re:Fast drilldown dashboards from a single Parquet file by nicoritschel on Monday August 24, 2026 @11:12PM (#49428673)
Perhaps changing is a better word.

Yes this was also my experience working at a large tech co. I work in fintech now and data volumes are low enough to maintain 2-3 minute up to a few hour data freshness.

Re:Fast drilldown dashboards from a single Parquet file by plaingrid on Tuesday August 25, 2026 @10:32AM (#49434866)
I have seen this on the business side too. A lot of we need live data requests don’t really need live data. If the decision is made once a day, hourly data is more than enough. Yet we still build dashboards that refresh every few minutes just because people asked for it.
Re:Fast drilldown dashboards from a single Parquet file by nicoritschel on Tuesday August 25, 2026 @07:12PM (#49441969)
In this case, we do actually need live data! We use wide events for monitoring business events and it lets us understand our platform with utmost flexibility, which is important for us as a fintech company.

Traditional observability is ill-suited for observability around business events. What if you forget to instrument a counter or gauge for something? In my experience it's far easier to log wide events with as much context as possible instead of agonizing over anticipating the dimensionality of metrics upfront (you're going to miss something).