News for hackers. Stuff that matters.

A comment on Windows-Use: an AI agent that interacts with Windows at GUI layer. Parent comment

Re:Windows-Use: an AI agent that interacts with Windows at G... by freedomben on Friday September 12, 2025 @01:07PM (#45224254)
Agreed. I've noticed ChatGPT when parsing screenshots writes out some Python code to parse it, and at least in the tests I've done (with things like, "what is the RGB value of the bullet points in the list" or similar) it ends up writing and rewriting the script five or so times and then gives up. I haven't tried others so I don't know if their approach is unique or not, but it definitely feels really fragile and slow to me
[ Parent | on HN ]
Re:Windows-Use: an AI agent that interacts with Windows at G... by Juminuvi on Friday September 12, 2025 @06:58PM (#45227723)
I noticed something similar. I asked it extract a guid from an image and it wrote a python script to run ocr against it...and got it wrong. Prompting a bit more seemed to finally trigger it to use it's native image analysis but I'm not sure what the trick was.
[ Parent | on HN ]
Re:Windows-Use: an AI agent that interacts with Windows at G... by morkalork on Friday September 12, 2025 @09:00PM (#45228394)
I've run into this with uploading audio and text files, have to yell at it to not write any code and use it's native abilities to do the job.
[ Parent | on HN ]
Re:Windows-Use: an AI agent that interacts with Windows at G... by spacebacon on Sunday September 14, 2025 @02:51AM (#45238005)
Probably just ask it to use native image analysis versus writing code. I have done this before extracting usernames from screenshots.
[ Parent | on HN ]