GPT uses 1024 tokens context window which does work out to a fair amount (given the massive vocab of 50k+ which means a token can be more than a word), though of course it's pretty limited.

Google's recent Reformer[0] allows you to do attention much more cheaply and I'm currently training a Reformer model that isn't quite as big but has a context of ~64k tokens (though a much smaller vocab). I'm not completely sure if this is the solution but it looks like a step in that direction and so far the model is doing pretty good (I also plan to post the weights when I'm finished, though I am not sure if Google don't just plan to do that themselves).

I am somewhat disappointed they went with just 1024 for this model, too though.

0. https://ai.googleblog.com/2020/01/reformer-efficient-transfo...