> why there's still so much work (and hacks and workarounds) on transformer to make it go faster and consume less VRAM whereas the same effort could be spent on other architectures

Probably because the pace of development is now no longer determined by ML researchers but by CS engineers. ML is "easy to pick up" until you have to understand the reasons certain architectures work or don't for different applications.