They start with some basic skills but the model builds the more complex skills. The specific skills that they start with depends on which task, for example RDR2 they start with only how to turn, move forward, shoot, and select an item from the item equip wheel and then a couple composite skills that were not _necessary_ but saved a lot of time - for example, "follow npc", since the model could follow an npc without it but it required a huge amount of back and forth. However, a lot of skills were actually learned by the model from the in-game tutorial in RDR2 explaining how to do things in-game!

> So this approach can't be generalised, can it?

The paper definitely demonstrates that the _approach_ can be generalized because they use the same approach across a variety of different environments and tasks, but you can also see that they did have to specialize the prompts, tools (like how the incoming screenshot was decorated with object detection / segmentation, stuff like that), and set of skills for each environment.

Right ... the output isn't really "keyboard & mouse operations" as stated in the abstract. The output is a sequence of atomic commands ("shoot", "select_weapon") or composite commands ("follow") that are then translated into mouse actions by a human-written layer.

Similarly, when I'm typing this post my output isn't "keyboard interrupts" nor "TCP frames". Those things do happen, but only by the grace of others' work.

Still I like the basic framework idea & I suppose future work looks at pushing the provided layer lower & lower - so the system can learn those atomic commands, rather than having humans provide them - and pushing the logic higher & higher via AI-generated composite commands.

Many games do happen to be set up this way wrt their input systems if they happen to leverage Steam input. Player actions in game are defined as abstract verbs with concrete input device binding being left to the player / underlying system. Valve is trying to nudge games towards being platonic objects isolated from specific systems (so they'll be readily compatible steam deck, vr, pc etc...). Translating these verbs in an automated way with an agent is an interesting side effect.

The problem is not triggering the action, the problem is being at the right place at the right time and looking into the right direction.

I think they seed it with some skills and then can generate more? The website allures to "skill generation" so arguably they are doing some level of code generation.