"However, instead of bloated bureaucracies and buildings that are unmaintainable at lower budgets, why didn't at least some colleges invest the proceeds into index funds to directly defray the costs to students, like at Yale?"

Investing implies that the tuition increases are not to make up for current budget shortfalls. This is often not the case. Many states, for example, cut the budget for higher education in 2008-2009 by the high double digits, with no real way for the universities in question to make that up.

"Or allocate to competitively recruit and retain proven professors with observable education outcomes (or invest in establishing metrics to identify such outcomes)?"

Honestly, because if they were doing it strategically to make up for the shortfall, they'd be competitively recruiting and retaining proven professors with a track record of funded research projects. Which is what they're doing, but that too is becoming more difficult as federal research budgets stagnate.