OSes do provide accessibility APIs - and usually if you use one of the more supported graphic toolkits there are somewhat easy ways to integrate directly in semantic ways, without needing to OCR and do fortune telling.

Usually it means that you need a deferred mode semantic tree of the application for accessibility UI to walk through, though.