Wow, this looks amazing.

The authors have developed Cradle, a multimodal-LLM-powered agent framework with six modules: Information Gathering, Self-Reflection, Task Inference, Skill Curation, Action Planning, and Memory. Once Cradle has processed high-level instructions, its inputs are sequences of computer screenshots. Its output is executable code for low-level keyboard and mouse control, enabling Cradle to interact with any software and complete long-horizon complex tasks without relying on any built-in APIs:

       Oversimplified Big-Picture Diagram

                 +------------+
                 |   Cradle   |    executable code
  screenshots -> |(high-level | -> for controlling
                 |  planning) |    keyboard & mouse
                 +------------+

The authors' experiments show what to me looks like impressive generalization and performance across software applications, successfully operating daily software like Chrome and Outlook, and across commercial video games: It is able to follow the main storyline and complete 40-minute-long missions in Red Dead Redemption 2, create a city of a thousand people in Cities: Skylines, farm and harvest parsnips in Stardew Valley, and trade and bargain to make a profit in Dealer's Life 2.

There are of course many caveats -- the technology is still in its infant stage -- but still, I'm impressed at how quickly things are progressing.

We sure live in interesting times!