Supabase Storage: now supports the S3 protocol

Re:Supabase Storage: now supports the S3 protocol by jarpineh on Friday April 19, 2024 @07:09AM (#40085332)

Hi, a question, but first some background. I've been looking at solutions to store columnar data with versioning, essentially Parquet. But, I'd also like to store PDFs, CSVs, images, and such for our ML workflows. I wonder if now, that Supabase is getting better for data science DuckDB crowd, could Supabase be that one solution for all this?

[ Parent | on HN ]

Re:Supabase Storage: now supports the S3 protocol by kiwicopple on Friday April 19, 2024 @07:17AM (#40085388)

> Parquet. But, I'd also like to store PDFs, CSVs, images

yes, you can store all of these in Supabase Storage and it will probably "just work" with the tools that you already use (since most tools are s3-compatible)

Here is an example of one of our Data Engineers querying parquet with DuckDB: https://www.youtube.com/watch?v=diL00ZZ-q50

We're very open to feedback here - if you find any rough edges let us know and we can work on it (github issues are easiest)

[ Parent | on HN ]

Re:Supabase Storage: now supports the S3 protocol by jarpineh on Friday April 19, 2024 @07:32AM (#40085490)

Well, this is great news. I'll take "just works" guarantee any day ;)

We have yet to make a commitment to any one product. Having Postgres there is a big plus for me. I'll have to see about doing a test or two.

[ Parent | on HN ]

Re:Supabase Storage: now supports the S3 protocol by Bnjoroge on Friday April 19, 2024 @06:15PM (#40092517)

you should look at lance(https://lancedb.github.io/lance/)

[ Parent | on HN ]