This technique of using distributed storage for large joins instead of shuffling between compute nodes also helps make your job robust to spot instance kills. Until disaggregated shuffle services are widely adopted, it can be really handy.