News for hackers. Stuff that matters.

A comment on Launch HN: GPT Driver (YC S21) – End-to-end app testing in natural language. Parent comment

Re:Launch HN: GPT Driver (YC S21) – End-to-end app testing i... by chairhairair on Wednesday October 23, 2024 @10:45AM (#41925564)
It's just a dream then.

It's completely at-odds with the strengths of LLMs (fuzzy associations, rough summaries, naive co-thinking).

[ Parent | on HN ]
Re:Launch HN: GPT Driver (YC S21) – End-to-end app testing i... by yorwba on Wednesday October 23, 2024 @10:58AM (#41925715)
Fuzzy associations seem relevant? Interact with the UI based on what it looks like, not the specific implementation details.
[ Parent | on HN ]
Re:Launch HN: GPT Driver (YC S21) – End-to-end app testing i... by chairhairair on Wednesday October 23, 2024 @11:19AM (#41925932)
No. Both of the requirements "to interact" and "based on what it looks like" require unshakable foundations in reality - which current models clearly do not have.

They will inevitably hallucinate interactions and observations and therefore decrease reliability. Worse, they will inject a pervasive sense of doubt into the reliability of any tests they interact with.

[ Parent | on HN ]
Re:Launch HN: GPT Driver (YC S21) – End-to-end app testing i... by tomatohs on Wednesday October 23, 2024 @01:11PM (#41927111)
> unshakable foundations in reality

Yes, you are correct that it entirely lays in the reputation of the AI.

This discussion leads to interesting question, which is "what is quality?"

Quality is determined by perception. If we can agree that an AI is acting like a user and it can use your website, we can assume that a user can use your website and therefor it is "quality".

For more, read "Zen and the Art of Motorcycle Maintenance"

[ Parent | on HN ]