The OS has additional information including how different graphics layers are composited, and what accessibility metadata is attached to interface elements. It out to be useful to exploit this to do better than screenshot parsing.