MLLMs are surprisingly bad at this out of the box and to some extent even with fine tuning. https://jina.ai/news/the-what-and-why-of-text-image-modality...

Loving this! Very surprising that the LLMs of today are so bad at understanding interfaces but it also makes it a very interesting case for finetuning!

Love that you made an interactive app to visualize how the model performs similar to how Meta AI usually releases their models

(direct link: https://qa-tech-minicpm-demo.gptengineer.run)

Did you use AI to generate this app?

yeah, we used https://gptengineer.app/ to create the playground! highly recommended

It’s fascinating to see how multimodal LLMs are being applied to UI understanding—definitely a challenging area that highlights the current limitations and potential of these models. The interactive app looks like a great way to visualize the model's performance in real-time, and it’s exciting to see tools like GPT Engineer being used to build such innovative demos. This kind of hands-on experimentation is essential for advancing the field and discovering new ways to improve model accuracy with fine-tuning. Looking forward to seeing where this goes!