I think the biggest issue is that those language models are still very limited by transformer window (2k tokens, each word usually consists of 3 tokens), and there is no visible improvement over this.

Your problem, code base don't fit into 700 words, you have no luck.