I think usefulness is a different question than capabilities. "LLMs can't do X" is an interesting claim, and it takes some level of systematic work to make it. If the capability exists at all, this is interesting, even if it's not useful. That being said, it doesn't mean such an existing capability will not prove useful. If something is fallible, it might be above a threshold worth using, it might be improved by doing the same techniques that are already being used, or it might be improved by adding on some sort of system to filter or refine responses.

> you can only know that an answer is correct when you already know it, and that's not very useful at all.

This statement makes sense on the surface, but I don't think it's true. All the time I get told things that I don't know the answer to beforehand, but judge pretty well whether it's right or not.