The OS has additional information including how different graphics layers are composited, and what accessibility metadata is attached to interface elements. It out to be useful to exploit this to do better than screenshot parsing.

Very cool work. Accurate GUI text and element parsing is exactly the kind of input that LLMs need to be effective agents.

This is not the intended use but it good working on parsing document layout from image.

One ponders the connections with the Recall feature