Accessibility interfaces are ideal for this situation, allowing an LLM to interact with a GUI strictly through text. Unfortunately, they're often an afterthought in implementations.