News for hackers. Stuff that matters.
Re:DuckDB is probably the most important geospatial software... by vasco on Sunday May 04, 2025 @01:00AM (#43884599)
I haven't yet understood this pattern (and I tried using duckdb). Unless you're only ever going to query those files once or twice in your life, importing them into postgres shouldn't be that long and then you can do the same or more than with DuckDB.

Also as a side note, is everyone just using DuckDB in memory? Because as soon as you want some multiple session stuff I'd assume you'd use DuckDB on top of a local database, so again I don't see the point but I'm sure I'm missing something.

Re:DuckDB is probably the most important geospatial software... by wenc on Sunday May 04, 2025 @02:12AM (#43884882)
> importing them into postgres shouldn't be that long and then you can do the same or more than with DuckDB.

Usually new data is generated regularly and would require creating a separate ETL process to ingest into Postgres. With DuckDB, no ETL is needed. New Parquet files are just read off the disk.

> Also as a side note, is everyone just using DuckDB in memory?

DuckDB is generally used as a single-user, and yes in-memory use case is most common. Not sure about use cases where a single user requires multiple sessions? But DuckDB does have read concurrency, session isolation etc. I believe write serialization is supported in multiple sessions.

With Parquet files, it's append-only so the "write" use-cases tend to be more limited. Generally another process generates those Parquet files. DuckDB just works with them.

Re:DuckDB is probably the most important geospatial software... by indeyets on Sunday May 04, 2025 @04:35AM (#43885400)
> Usually new data is generated regularly

This part was not obvious. In a lot of cases geodata is mostly stable and reads/searches dominate over appends. And that’s why we keep this in DB (usually postgis, yes).

So DuckDB is optimised for very different use case and it is not always obvious when it’s mentioned

Re:DuckDB is probably the most important geospatial software... by simlevesque on Sunday May 04, 2025 @06:11AM (#43885781)
But DuckDB works just as well with static data.
Re:DuckDB is probably the most important geospatial software... by wenc on Sunday May 04, 2025 @11:35AM (#43887311)
This is the more trivial use case (static data, heavy reads) and DuckDB is absolutely optimized for this use case.

DuckDB also provides a vectorized, parallelized engine. When I run a query all of my 32 cores light up on too.

Re:DuckDB is probably the most important geospatial software... by lugarlugarlugar on Sunday May 04, 2025 @01:58AM (#43884823)
I think the main point is not having to store a duplicate of the 600GB of input data.