Re:Preview: Amazon S3 Tables and Lakehouse in DuckDB
by pletnes on Tuesday March 18, 2025 @03:13PM
(#43403646)
I have the same thoughts. However my impression is also that most orgs would choose eg databricks or something for the permission handling, web ui, ++ so what is the equivalent «full rig» with duckdb and S3 / blob storage?
Re:Preview: Amazon S3 Tables and Lakehouse in DuckDB
by RobinL on Tuesday March 18, 2025 @03:39PM
(#43404007)
Yeah I think that's fair, especially from the 'end consumer of the data' point of view, and doing things like row-level permissions.
For the ETL side, where often whole-table access is good enough, I find Spark in particular very cumbersome - there's more than can go wrong vs. DuckDB and it's harder to troubleshoot.
Re:Preview: Amazon S3 Tables and Lakehouse in DuckDB
by hn1986 on Tuesday March 18, 2025 @03:07PM
(#43403556)
from the blog: "This is a very interesting new development, making DuckDB potentially a suitable replacement for lakehouse formats such as Iceberg or Delta lake for medium scale data."
I don't think we'll ever see this, honestly.
excellent podcast episode with Joe Reis - I've also never understood this whole idea of "just use Spark" or you gotta get on Redshift.
Re:Preview: Amazon S3 Tables and Lakehouse in DuckDB
by teruakohatu on Tuesday March 18, 2025 @07:49PM
(#43406668)
> excellent podcast episode with Joe Reis - I've also never understood this whole idea of "just use Spark" or you gotta get on Redshift.
Can you link to the podcast episode?
Re:Preview: Amazon S3 Tables and Lakehouse in DuckDB
by hn1986 on Tuesday March 18, 2025 @10:50PM
(#43407828)
episode and transcript Referenced in the blog: https://www.robinlinacre.com/recommend_duckdb/#:~:text=in%20...
Re:Preview: Amazon S3 Tables and Lakehouse in DuckDB
by yakshaving_jgt on Tuesday March 18, 2025 @02:27PM
(#43402959)
Funny, I read TFA and came to the comments to share exactly this recent blog post of yours. Big fan of your work, Robin!
May be of interest to people who:
- What to know what DuckDB is and why it's interesting
- What's good about it
- Why for orgs without huge data, we will hopefully see a lot more of 's3 + duckdb' rather than more complex architectures and services, and hopefully (IMHO) less Spark!
https://www.robinlinacre.com/recommend_duckdb/
I think most people in data science or data engineering should at least try it to get a sense of what it can do
Really for me, the most important thing is it makes it so much easier to design and test complex ETL because you're not constantly having to run queries against Athena/Spark to check they work - you can do it all locally, in CI, set up tests, etc.