OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computers

Re:OSWorld: Benchmarking Multimodal Agents for Open-Ended Ta... by mhuffman on Sunday April 28, 2024 @09:19PM (#40193426)

I am not an AGI stan, but it is clear to me that this gap will close substantially, and quickly ... like 6 months or so. The break-neck speed of development and (to me) the counterintuitive zeal with which open source developers are just donating tools and code that are going to hamstring them somewhat pretty soon is breathtaking. Think about how "most" companies use tech. Most companies are not big, they have a minimum set of tools to get what they need done. They currently, and routinely, have human mistakes. These are the types of tasks that agents trained by human demonstration can do soon. The only one that I suspect will be slow for adoption is bookkeeping, because you could theoretically end up on the wrong side of the IRS with mistakes.

[ Parent | on HN ]

Re:OSWorld: Benchmarking Multimodal Agents for Open-Ended Ta... by ec109685 on Sunday April 28, 2024 @09:59PM (#40193638)

This is a thoughtful article on the current limitations of LLM and agents: https://www.strangeloopcanon.com/p/what-can-llms-never-do

A good example is that you can teach a first grader how to play wordle, but an LLM can’t be taught.

That’s a pretty big gap to a situation where most humans can be replaced.

[ Parent | on HN ]

Re:OSWorld: Benchmarking Multimodal Agents for Open-Ended Ta... by mhuffman on Sunday April 28, 2024 @10:04PM (#40193664)

Except that the problems most (and by that I mean non-software, non-billion-dollar) businesses have are not wordle type puzzle problems. They are 'go through all of our invoices and put xxxx data in a spreadsheet for me please' type problems or 'copy the data from this app and put it in that app' type problems or responding to emails or responding to simple customer support questions. And these are definitely in the wheelhouse of LLM near-future capabilities.

[ Parent | on HN ]

Re:OSWorld: Benchmarking Multimodal Agents for Open-Ended Ta... by bitwize on Sunday April 28, 2024 @10:21PM (#40193748)

And the human performers of many of these tasks have been replaced not by AIs but by VBA scripts. It's almost a trope that many Hackernews' first jobs were something of this type, and on the first day they wrote a VBA script to do most or all of it and then collected pay for, effectively, playing Minesweeper all day.

[ Parent | on HN ]

Re:OSWorld: Benchmarking Multimodal Agents for Open-Ended Ta... by mhuffman on Sunday April 28, 2024 @10:39PM (#40193831)

This is true, and how I got into software development as a job, but going forward they will likely be replaced by AI agents off of some site like Zapier.

[ Parent | on HN ]