News for hackers. Stuff that matters.
Re:Interesting ideas in Observable Framework by orf on Sunday March 03, 2024 @07:28PM (#39585837)
This looks fantastic! I’ve been waiting for something like this.

My only gripe is that data loaders don’t seem to support Parquet files, which is really annoying.

There’s an interesting possibility here where you can have large datasets in Parquet, exposed via HTTP, whilst being generated at build time with all the benefits that gives you (being able to read only specific columns, filtering via row group statistics etc). Not dissimilar to the “SQLite over http” WASM demo I guess.

Because right now I need to take my large, nicely compressed dataset and export it as either a CSV or a zip file? And then the browser needs the entire thing, even if I’m just viewing a subset of the data? Which is much bigger and much slower than it needs to be.

Re:Interesting ideas in Observable Framework by simonw on Sunday March 03, 2024 @08:06PM (#39586054)
There's a parquet example here:

https://github.com/observablehq/framework/blob/main/examples...

Using data from here: https://github.com/observablehq/framework/tree/main/examples...

Rendered version here: https://observablehq.com/framework/examples/api/

If you want to run a Data loader that outputs to parquet there are plenty of ways to do that - I would suggest a Bash or Python script that wraps DuckDB.

Re:Interesting ideas in Observable Framework by mbostock on Monday March 04, 2024 @01:25AM (#39587573)
We do support it. (And use it!) Please see: https://observablehq.com/framework/lib/arrow#apache-parquet
Re:Interesting ideas in Observable Framework by orf on Monday March 04, 2024 @03:06AM (#39588090)
Ahh, amazing! That’s not entirely clear from the data loading docs[1], which when I read it seemed only focused around CSV and JSON.

1. https://observablehq.com/framework/loaders