Re:Using GPT-4 Vision with Vimium to browse the web
by sunshadow on Thursday November 09, 2023 @06:10AM
(#38203550)
But its too expensive to become practical with the OpenAI API. Also, demo is cool until you see the real-world webpages, then you'll realize that this only works less than %50 of webpages.
Re:Using GPT-4 Vision with Vimium to browse the web
by og_kalu on Thursday November 09, 2023 @07:34AM
(#38204192)
GPT-4V may be surprisingly robust here. Set of mark prompting(which is accomplished here with Vim) improves grounding by a silly high amount.
https://som-gpt4v.github.io/
Re:Using GPT-4 Vision with Vimium to browse the web
by abrichr on Friday November 10, 2023 @07:49AM
(#38218194)
Agreed! This is part of the motivation behind https://github.com/OpenAdaptAI/OpenAdapt
I think Adept pivoted along the way but original concept was very similar to this.