News for hackers. Stuff that matters.
Re:Cradle: Empowering Foundation Agents Towards General Comp... by finnh on Friday July 12, 2024 @10:15AM (#40945780)
Right ... the output isn't really "keyboard & mouse operations" as stated in the abstract. The output is a sequence of atomic commands ("shoot", "select_weapon") or composite commands ("follow") that are then translated into mouse actions by a human-written layer.

Similarly, when I'm typing this post my output isn't "keyboard interrupts" nor "TCP frames". Those things do happen, but only by the grace of others' work.

Still I like the basic framework idea & I suppose future work looks at pushing the provided layer lower & lower - so the system can learn those atomic commands, rather than having humans provide them - and pushing the logic higher & higher via AI-generated composite commands.

Re:Cradle: Empowering Foundation Agents Towards General Comp... by fordacious on Friday July 12, 2024 @11:32AM (#40946506)
Many games do happen to be set up this way wrt their input systems if they happen to leverage Steam input. Player actions in game are defined as abstract verbs with concrete input device binding being left to the player / underlying system. Valve is trying to nudge games towards being platonic objects isolated from specific systems (so they'll be readily compatible steam deck, vr, pc etc...). Translating these verbs in an automated way with an agent is an interesting side effect.
Re:Cradle: Empowering Foundation Agents Towards General Comp... by miohtama on Friday July 12, 2024 @11:46AM (#40946642)
The problem is not triggering the action, the problem is being at the right place at the right time and looking into the right direction.