News for hackers. Stuff that matters.
Re:Apache Arrow and MinIO by bwc150 on Monday September 21, 2020 @03:10PM (#24547194)
The storage costs are only a tiny fraction of the costs involved with accessing and analyzing the data using snowflake "credits"
Re:Apache Arrow and MinIO by georgewfraser on Monday September 21, 2020 @03:47PM (#24547628)
For sure, but you're not going to fix that by making your own data lake using, for example, Parquet-on-S3. You're still going to pay the cost of compute when you analyze that data, and a well-optimized commercial database system is extremely hard to beat. Even if you look at Presto, and you exclude the people costs of managing it yourself, it still can't beat the commercial systems: https://fivetran.com/blog/warehouse-benchmark
Re:Apache Arrow and MinIO by hodgesrm on Tuesday September 22, 2020 @12:23AM (#24550824)
That's because Snowflake happens to charge relatively little for backing storage at the moment. As I recall it's about the same as object storage. Virtual data warehouses on the other hand are quite expensive, especially if they have a lot of compute.