News for hackers. Stuff that matters.
Re:TaxyAI: Open-source browser automation with GPT-4 by mlboss on Tuesday March 28, 2023 @05:16PM (#35347998)
This projects demonstrates the death of UI. UIs were created for humans to interact with software not for bots to perform tasks. If all you need to provide is your text request then we don't need UIs. All softwares can just expose rest/text interfaces and LLM can perform the task for you.
Re:TaxyAI: Open-source browser automation with GPT-4 by Zetobal on Tuesday March 28, 2023 @06:26PM (#35348869)
You won't cut off your legs just because you drove a car for the first time.
Re:TaxyAI: Open-source browser automation with GPT-4 by RC_ITR on Tuesday March 28, 2023 @06:49PM (#35349171)
But certain countries will build a culture and infrastructure that completely revolves around the car and results in the average citizen walking only 3,000 steps per day [0].

Interfaces will exist because sometimes we need them (walking between your car and the actual destination) and sometimes we like them (just like some people like walking), but just like walking, they may lose massive share.

[0]https://www.mayoclinic.org/healthy-lifestyle/fitness/in-dept....

Re:TaxyAI: Open-source browser automation with GPT-4 by anigbrowl on Thursday April 06, 2023 @06:46PM (#35475551)
Not the best analogy considering that most cars use pedals.
Re:TaxyAI: Open-source browser automation with GPT-4 by eshnil on Tuesday March 28, 2023 @11:44PM (#35352152)
More like the death of APIs. If bots can handle UI designed for humans, there's no reason to build and deploy APIs.
Re:TaxyAI: Open-source browser automation with GPT-4 by riku_iki on Tuesday March 28, 2023 @06:26PM (#35348861)
Many actions can be made faster and with better precision in UI compared to slow texting with gpt.
Re:TaxyAI: Open-source browser automation with GPT-4 by mbil on Tuesday March 28, 2023 @07:20PM (#35349501)
This is true today, but will it be true in the future where language models are a common interface between human language (probably voice) and software actions?

When I go to my bank's website, it's easy for me to find my tax forms. They're usually one or two clicks away in some prominent top-level navigation component. But I would rather converse: "Download tax forms"/"Which ones: A, B, or C"/"The first two"/"Here ya go".

And in the future I'd rather say "Hey computer, download my tax forms from my bank and attach them in a draft email to <tax preparer>".

And in the later future that I'd rather say "Hey computer, do my taxes", and it will know what sources to gather info from and how to make sense of the numbers and how to file on my behalf. Or better yet, the machine will anticipate I need my taxes done and will kick off that process and solicit approvals and information from me. (Maybe by then the government will just tell me how much I owe ;))

Power users will likely sometimes find utility in dropping down "close to the metal" -- i.e., pointing and clicking and typing. But power users will mostly be composing workflows of models working together to achieve tasks (the term "scripts" will fit nicely). Yes this will be buggy and error-prone, but there will be glue models papering over the errors and trying different things and waiting for outages until the human user's intent is done.

Language models are giving us another layer of abstraction over software. Perhaps the final layer, as this one can interact directly with language (which is reified thought). This layer has the ability to cope with the inherent ambiguity of thought by being conversational -- it can ask you to clarify what you mean and gain confidence that it understands your intent.

Re:TaxyAI: Open-source browser automation with GPT-4 by pragmatick on Wednesday March 29, 2023 @03:38AM (#35353869)
Creating the actions may be easier but parsing / understanding them will often be easier in any format that prose / plain text. I'd rather have a UI with a list of steps and colors and what not than a text I have to read.

Take for example the list of steps in the project. They contain a lot of redundant information I have to mentally ignore.

Re:TaxyAI: Open-source browser automation with GPT-4 by riku_iki on Tuesday March 28, 2023 @08:11PM (#35350054)
> When I go to my bank's website, it's easy for me to find my tax forms. They're usually one or two clicks away in some prominent top-level navigation component. But I would rather converse: "Download tax forms"/"Which ones: A, B, or C"/"The first two"/"Here ya go".

I think this supposedly should be solved by search box on the site, where you enter: "tax forms" and got results with download button.

Re:TaxyAI: Open-source browser automation with GPT-4 by 93po on Tuesday March 28, 2023 @07:38PM (#35349699)
Yes, but I would rather the AI determine which actions these are and dynamically generate UI elements for me instead of relying on whatever shithead UI/UX person at LinkedIn wants me to look at to trick me into sharing all of my contact book.
Re:TaxyAI: Open-source browser automation with GPT-4 by riku_iki on Tuesday March 28, 2023 @08:09PM (#35350038)
I think era of shithead AI is also coming.
Re:TaxyAI: Open-source browser automation with GPT-4 by rurp on Tuesday March 28, 2023 @10:14PM (#35351298)
Yep. The notion that AIs, funded by companies like Microsoft of all places, is going to lead to a golden age of user interfaces boggles my mind.
Re:TaxyAI: Open-source browser automation with GPT-4 by 411111111111111 on Wednesday March 29, 2023 @02:00AM (#35353216)
To be fair, didn't they actually do that with the UI element it's named after (windows...)?

I think the menu bar, the window with the header bar and the action buttons where all pioneered by them back then, or am I mistaken?

It's been 30 yrs though, so I get what you mean.

Re:TaxyAI: Open-source browser automation with GPT-4 by Jorengarenar on Wednesday March 29, 2023 @12:38PM (#35359721)
IIRC, Microsoft "borrowed" the design from Apple Macintosh, which in turn was based on Xerox's technology. So no, they weren't the pioneers.
Re:TaxyAI: Open-source browser automation with GPT-4 by rurp on Wednesday March 29, 2023 @10:19PM (#35366980)
That's fair, I think the Microsoft of 30 years ago did pretty well with UIs. That said, given the state of Windows 11 and their other modern products I think their current priorities are quite a bit different.
Re:TaxyAI: Open-source browser automation with GPT-4 by evilduck on Tuesday March 28, 2023 @09:19PM (#35350743)
It was trained on the internet. Shithead AI is already lurking beneath the veneer.
Re:TaxyAI: Open-source browser automation with GPT-4 by swexbe on Wednesday March 29, 2023 @02:34AM (#35353427)
User: Turn off email notifications for LinkedIn.

ShitGPT: Are you sure you want to turn them off? LinkedIn notifications helpfully provide you with the latest news about your workplace and collogues past and present.

User: Yes, turn them off.

ShitGPT: Did you know that research has proven that employees with LinkedIn notifications turned on earn 10% higher salaries compared to their peers? Do you still want to to off notifications?

User: Yes.

ShitGPT: What is your reason for wanting to turn off notifications? Your answer has to be at least 400 words long.

...

ShitGPT: No, the words 'fuck openai' copied 400 times is not a valid answer.

...

ShitGPT: Error Too many requests in 1 hour. Try again later (of course you'll have to start over, tee hee)

Re:TaxyAI: Open-source browser automation with GPT-4 by 93po on Wednesday March 29, 2023 @11:52AM (#35359006)
Re:TaxyAI: Open-source browser automation with GPT-4 by alignment on Tuesday March 28, 2023 @08:47PM (#35350446)
Perhaps the death of UIs for doing basic tasks. UIs for visualizing data will remain for some time.
Re:TaxyAI: Open-source browser automation with GPT-4 by bryanrasmussen on Wednesday March 29, 2023 @01:59AM (#35353210)
This project demonstrates the need to automate things that owners of sites don't want you to automate.
Re:TaxyAI: Open-source browser automation with GPT-4 by manojlds on Wednesday March 29, 2023 @04:41AM (#35354304)
What is the equivalent of captcha in this world
Re:TaxyAI: Open-source browser automation with GPT-4 by CSSer on Wednesday March 29, 2023 @10:39AM (#35357943)
I seem to recall people joking about the irony of bots solving captchas a few years back. No one's laughing now.
Re:TaxyAI: Open-source browser automation with GPT-4 by jerryzh on Thursday March 30, 2023 @10:54AM (#35373563)
There was a fascinating (but sadly discontinued) app called Shortcat for MacOS. It basically let you control your entire MacOS with keyboard typing texts.

Thus, this can actually be done at larger scale for over 10 years.

Update: It turns out the development is resumed again, but I don't use mac anymore.

Re:TaxyAI: Open-source browser automation with GPT-4 by flarn2006 on Sunday April 02, 2023 @03:08AM (#35407963)
From CLI to GUI and back again.