This is akin to an analytic statement, eg, "all bachelors are married". The truth is completely within the definition of the statement. Compare this to a synthetic statement such as "it is raining outside". In this case the truth is contingent on facts outside of the statement itself.
When LLMs are faced with an analytic statement they are more reliable. When they are faced with a synthetic statement they are prone to confabulate and are unreliable.
Hrm. I've found LLM summaries to be of... dubious reliability. When someone posts an article on this here orange website, these days someone will sometimes 'helpfully' post a summary generated by a magic robot. Have a look at these, sometime. They _often_ leave out key details, and sometimes outright make shit up.
Interesting article on someones' experiences with this recently: https://ea.rna.nl/2024/05/27/when-chatgpt-summarises-it-actu...
Original article: https://osa1.net/posts/2024-10-09-oop-good.html
LLM (ChatGPT o1-preview) results: https://chatgpt.com/share/67166301-00d8-8013-9cf5-e8a980aca7...
LGTM!
I'd like to know which model is used in the article you've referenced as well as the prompt. I also suspect that 50 pages is pushing up to the limits of the context window and has an impact on the results.
---
As for the article itself... Use a language like F# or OCaml and you get a functional-first language that also supports OOP!
So, your problem there is 'reliable'. LLMs, fairly fundamentally, cannot do 'reliable'. If you're looking for reliable, you likely are looking at a different tech entirely.