I'm ambivalent. It summarises text on papers that are simple enough, but interestingly it fails in ways that suggests it sometimes fails because it isn't "smart enough." E.g. I asked it about "Code Generation on the Fly: A Key to Portable Software", the doctoral dissertation of Professor Michel Franz, and a paper I know very well, and it spit back a description of the process that was perhaps superficially correct but the kind of correct you'd expect of a lazy student who'd not bothered to actually understand the paper. The summary was largely meaningless twaddle that didn't actually tell you anything useful.

I then started probing it for more detail, and pointing out mistaken assumptions, and it got better. But it was first when I pointed out that the technique for encoding and decoding has a resemblance to Lempel-Ziv-Welch compression that it succeeded in identifying the key part of the paper and give a description that while still not quite accurate at least captured the essence of the method.

It felt like I was giving an exam on the topic rather than "getting help" summarising it, and if I didn't know the paper, it would've been hard to tell if I was getting close without reading it.

For conceptually simpler things, though, I've gotten great results, and it seems to work great for things where I know what a good result looks like ahead of time, and "just" want to save looking up details where I'll recognise the right thing when I see it but don't remember it by heart.

That said, to be able to point out the resemblance to LZW and have it connect the dots is at the same time fairly impressive - I've had to point that out to experienced developers (the paper mentions Welch, "This method bears some resemblance to commonly used data compression schemes [Wel84]", with Wel84 reading: "T. A. Welch; A Technique for High_Performance Data Compression; IEEE Computer, 17:6, 8-19; 1984. {2}" but it does not go into any further detail of how they relate).

I think, overall, that this is an indication that it will get better at this as the model size increases further. It was able to at least recognise the semantic similarity or identify that this similiarity has been described elsewhere and use it to guide it's response in a way that strikes me as far from trivial.

[It did go on to hallucinate several relevant papers on the subject; interestingly it confirmed it had made them up when I asked if it had]