News for hackers. Stuff that matters.
Re:DuckDB is probably the most important geospatial software... by touisteur on Sunday May 04, 2025 @06:18AM (#43885814)
And now I'm curious whether there's a way to actually index external files (make these queries over 600GB faster) and have this index (or many indices) be persistent. I might have missed that when I looked at the docs...
Re:DuckDB is probably the most important geospatial software... by wenc on Sunday May 04, 2025 @01:57PM (#43888252)
If the data is in Parquet they are already indexed in a sense. No further indexing necessary.

If they are stored in DuckDB’s native format (which I don’t use), it supports some state of the art indices.

https://duckdb.org/docs/stable/sql/indexes.html

I find Parquet plenty fast though.

Re:DuckDB is probably the most important geospatial software... by touisteur on Tuesday May 06, 2025 @01:38AM (#43902109)
Ah thanks, of course. I was thinking of dealing with millions of (Geo)JSON files adding up to terabytes, without copying/duplicating them though, mostly indexing. I used to do that with postgres foreign data wrappers and had hopes for duckdb :-). But that's a question for SO or other forum.