Re:Show HN: Tarsier – Vision utilities for web interaction a...
by jackienotchan on Wednesday May 15, 2024 @04:01PM
(#40371651)
Looking at OpenAdapt, I'm wondering why they didn't integrate Tarsier into AgentGPT, which is their flagship github repo but doesn't seem to be under active development anymore.
In https://github.com/OpenAdaptAI/OpenAdapt/blob/main/openadapt... we use FastSAM to first segment the UI elements, then have the LLM describe each segment individually. This seems to work quite well; see https://twitter.com/OpenAdaptAI/status/1789430587314336212 for a demo.
More coming soon!