News for hackers. Stuff that matters.
Re:Using GPT-4 Vision with Vimium to browse the web by bboygravity on Thursday November 09, 2023 @02:10AM (#38202035)
Funny that you and others on here don't seem to realize that literally everybody who uses the internet has the exact same data entry problem all the time. Blame it on "old software", but how about the entire internet?

copying (or in most cases even worse: re-typing) form data from one location on the screen into yet another webform.

Username, password, email address, physical address, credit card info etc etc.

Some extensions try to help with data entry, but none of them work properly and consistently enough to really help. Even consistently filling just username and pw is too much to ask.

It's my number 1 frustration when using the internet (worse than ads) and I find it mind-blowing that this hasn't been solved yet with or without LLMs.

I would pay a montly fee for any software that solves this once and for all and it sounds like it's coming (and I'm already paying their monthly fee).

Re:Using GPT-4 Vision with Vimium to browse the web by fragmede on Thursday November 09, 2023 @02:39AM (#38202184)
consistently filling out username and password is all I wanted from my password manager, but it turns out it handles credit card number and other bits of information for me as well.
Re:Using GPT-4 Vision with Vimium to browse the web by arkitaip on Thursday November 09, 2023 @03:30AM (#38202479)
I've used Bitwarden to faster fill out job applications.
Re:Using GPT-4 Vision with Vimium to browse the web by mewpmewp2 on Thursday November 09, 2023 @09:13PM (#38214272)
Doesn't chrome out of the box handle all of that?
Re:Using GPT-4 Vision with Vimium to browse the web by pseudosaid on Thursday November 09, 2023 @02:36AM (#38202173)
use a password manager. i havent copy pasted form data twice on a site in a long time
Re:Using GPT-4 Vision with Vimium to browse the web by TeMPOraL on Thursday November 09, 2023 @03:17AM (#38202379)
> It's my number 1 frustration when using the internet (worse than ads) and I find it mind-blowing that this hasn't been solved yet with or without LLMs.

Simple: it's because not solving this problem is how our godawful industry makes most of its money. Empowering the user means relinquishing control over their "journey"[0]. Ergonomics means fewer opportunities to upsell or show ads.

I don't have the link handy, but I'm reminded of one of the earliest Windows user interface guidelines documents, back from Windows 95/98 era, which, in a section about theming/visual style, already recognized that they have to allow for full flexibility, because vendors will insist on fucking the experience up for the sake of branding anyway, and resisting it is futile[1].

--

[0] - I'm trying really hard to hold back my contempt towards terms like this, and the whole salesy way of viewing human-computer interactions.

[1] - They put it in much more polite terms, but the feeling of helplessness was already there.

Re:Using GPT-4 Vision with Vimium to browse the web by itronitron on Thursday November 09, 2023 @05:24AM (#38203231)
>> because vendors will insist on fucking the experience up for the sake of branding anyway

I see that you too have at some point installed printer driver software.

Re:Using GPT-4 Vision with Vimium to browse the web by musha68k on Thursday November 09, 2023 @10:37AM (#38206348)
Ted Nelson’s “intertwingularity” isn’t far off from the data entry problem described. He argues for universal data access where duplication is obsolete. Imagine form data as a single, linkable object across the web, editable in one place, reflected everywhere—no re-typing, just seamless auto-fill. That’s the unrealized potential of hypertext.
Re:Using GPT-4 Vision with Vimium to browse the web by anonzzzies on Thursday November 09, 2023 @03:39AM (#38202533)
Yeah, my dream would be using this to scrape pages, pop the content into my provide db, serving it up in my own format (which is going to be a white page with letters with inline images and videos that are not ads. And my interactions fed back to the vision model to post in the original. So I never have to see a ‘design’ (heavy js riddled unreadable crap) again in my life. And so I can, with my own tooling, browse and reuse my history including content instead rely on all the broken stuff bolted on the web.
Re:Using GPT-4 Vision with Vimium to browse the web by loud_cloud on Thursday November 09, 2023 @04:16AM (#38202758)
FTL. See NiagraFiles.
Re:Using GPT-4 Vision with Vimium to browse the web by williamcotton on Thursday November 09, 2023 @04:20AM (#38202791)
Bash pipes? The free flow of information through composable tools.

The commercial web? Not the above.

This is just a baseline. I’m sure that an LLM can help the issue but the biggest problem is that these varied HTTP-with-datastores are islands passing messages in bottles back and forth while a bash pipeline is akin to fiber optics.