I agree that this is the part that needs more work, and is most uncertain. Increasing context windows seems like a fairly straightforward computational challenge (albeit potentially expensive). On the other hand, whether or not we can scale current models towards "true understanding" (or similar), is a total unknown atm.

I still think we will get useful things from scaling up current models though. I've already got a lot of value out of Copilot, for instance, and I'm looking forward to the next version based on GPT-4. Recently, I've been using the GPT-3 Copilot to write a lot of pandas/matplotlib code, which is fairly straightforward and repetitive, but as mainly a Java developer, I just don't have the APIs at my fingertips. Copilot helps a lot with this sort of thing.