> If I were independent I would recommend people buy and build on-prem clusters and shuffle data out of fast scratch into Glacier,

Pretty much 100% agree with this statement. Only exception is maybe at the beginning of a long series of computation, you might want to start on-demand to fully understand and size your exact needs, and then provision off-prem.