Re:Why GitHub Won
by timschmidt on Monday September 09, 2024 @07:42PM
(#41495543)
Accessibility interfaces are ideal for this situation, allowing an LLM to interact with a GUI strictly through text. Unfortunately, they're often an afterthought in implementations.
You could suggest using AI to OCR the canvas but even that would be subpar because most sites that use HTML provide multiple screens worth of info but sites that render to a canvas render only what's visible. The rest of the data is internal. You'd not only need the AI to successfully OCR the text, you'd need it to interact with the page get it to render what's not currently displayed.