Large Language Models Do Not Recognize Identifier Swaps in Python

Re:Large Language Models Do Not Recognize Identifier Swaps i... by KMnO4 on Sunday May 28, 2023 @10:28AM (#36104265)

My pet peeve is when articles say “LLMs can’t do X” but they really mean “OpenAI’s Chat GPT”.

It’s like saying “cars can’t run off gasoline” but their testing methodology is based around a Tesla.

[ Parent | on HN ]

Re:Large Language Models Do Not Recognize Identifier Swaps i... by TeMPOraL on Sunday May 28, 2023 @10:38AM (#36104354)

Except right now, GPT-4 is the best LLM in terms od performance/capabilities, by a large degree; every other one is strictly inferior, and they all share the same architecture and training methodology. So it's like comparing cars from 10 years ago against a top-of-the-line Toyota from 10 years ago and top-of-the line Toyota from today (or BMW, or whoever does quality cars; I'm not a car person).

[ Parent | on HN ]

Re:Large Language Models Do Not Recognize Identifier Swaps i... by gduchgdjv on Sunday May 28, 2023 @11:00AM (#36104581)

also it's hosted elsewhere. they may be updating it. they may redirect your calls to different buckets. some queries might perform worse as they are under load.

the whole "prompt science" is utter ridiculous to begin with because you can't even isolate what your testing. not to mention you're graduating with a paper on the 2000 equivalent of "how to find mp3s on Google" lol.

[ Parent | on HN ]