Yep, the number has been expanding. I just picked 512 because Bert uses that. But yeah 50k and beyond unlocks new ways of using context. I can’t wait to see how they optimize and improve the reformer model.

I expect there to be an improvement like there was for

BERT -> Albert