sfilipco

I couldn’t use Google Maps on Firefox on Mac yesterday. I couldn’t click on the search bar.

I agree with the theme of the article. My reply was to parent comment which has a 6 TB working set.

I agree that keeping data local is great and should be the first option when possible. It works great on 10GB or even 100GB, but after that starts to matter what you optimize for because you start seeing execution bottlenecks.

To mitigate these bottlenecks you get fancy hardware (e.g oracle appliance) or you scale out (and get TCO/performance gains from separating storage and compute - which is how Snowflake sold 3x cheaper compared to appliances when they came out).

I believe that Trino on HDFS would be able to finish faster than awk on 6 enterprise disks for 6TB data.

In conclusion I would say that we should keep data local if possible but 6TB is getting into the realm where Big Data tech starts to be useful if you do it a lot.