I agree - unfortunately we have gone so far down the rabbit hole optimizing transformer models.

Alternative models like S4 have been able to get transformer level performance with O(N) sequence length scaling.