Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku

Re:Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku by quantadev on Tuesday October 22, 2024 @12:59PM (#41916273)

I think the best use case for AI `Computer Use` would be a simple positioning of the mouse and asking for conformation before a click. For most use cases this is all people will want/need. If you don't know how to do something, it is basically teaching you how, in this case, rather than taking full control and doing things so fast you don't have time to stop of going rogue.

[ Parent | on HN ]

Re:Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku by luigipederzani on Tuesday October 22, 2024 @03:21PM (#41917720)

I totally agree with you. At orango.ai, we have implemented the auto-click feature, but before it clicks, we position the cursor on the button and display a brief loading animation, allowing the user to interrupt the process.

[ Parent | on HN ]

Re:Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku by quantadev on Wednesday October 23, 2024 @12:55AM (#41921950)

That's a great approach! User is still supervising but not having to do anything but watch, most of the time. That's perfect.

[ Parent | on HN ]

Re:Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku by accrual on Tuesday October 22, 2024 @02:01PM (#41916914)

Maybe we could have both - models to improve accessibility (e.g. for users who can't move their body well) and models to perform high level tasks without supervision.

It could be very empowering for users with disabilities to regain access computers. But it would also be very powerful to be able to ask "use Photoshop to remove the power lines from this photo" and have the model complete the task and drop off a few samples in a folder somewhere.

[ Parent | on HN ]

Re:Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku by quantadev on Tuesday October 22, 2024 @02:54PM (#41917442)

Yep. I agree. The "auto-click" thing would be optional. Should be able to turn it on and off. With auto-click off it would just position the mouse and say "click here".

[ Parent | on HN ]

Re:Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku by HappMacDonald on Tuesday October 22, 2024 @08:23PM (#41920200)

Cluade scans page and decides which button to click before the screen layout is finished. By the time user authorizes the click, layout has shifted and your click lands on malware advertisements.

[ Parent | on HN ]

Re:Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku by quantadev on Tuesday October 22, 2024 @10:23PM (#41921132)

lol. If any website ever did that to me it would be the last time I ever went to it. Not a big concern for me.

[ Parent | on HN ]

Re:Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku by HappMacDonald on Wednesday October 23, 2024 @03:53AM (#41922813)

Youtube constantly moves it's layout seconds after the page begins to paint, so I try to click on fullscreen or whatever and then the viewer shifts to the side and I wound up clicking a link to some other video.

Probably would have been an ad there if I didn't block those, though.

[ Parent | on HN ]

Re:Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku by EGreg on Tuesday October 22, 2024 @02:57PM (#41917479)

People would mostly just rubber-stamp it

But it would slow down the masses

Some people would jailbreak the agents though

[ Parent | on HN ]

Re:Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku by falcor84 on Tuesday October 22, 2024 @07:58PM (#41920015)

Even better, how about giving the AI the capability to move and draw and overlay on the screen with a separate virtual cursor as in a Zoom session?

[ Parent | on HN ]

Re:Computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku by quantadev on Wednesday October 23, 2024 @01:03AM (#41921981)

I like the drawing on the screen idea. The biggest use case of that I can think of is drawing a black rectangle over all ADs!!!!

[ Parent | on HN ]