News for hackers. Stuff that matters.
Re:R1 Computer Use by iiJDSii on Thursday February 06, 2025 @04:24PM (#42966640)
What does your perception look like, are you using raw screenshots? GUI snapshots? Vision is very difficult for these, and snapshots are incomplete, is what I've found in some earlier experiments.
Re:R1 Computer Use by mountainriver on Thursday February 06, 2025 @04:39PM (#42966765)
Perception is just 1-2 screenshots. A number of recent VLM models have a lot more pretraining data on GUI interactions, which helps.
Re:R1 Computer Use by iiJDSii on Thursday February 06, 2025 @04:47PM (#42966820)
Such as? Are they able to recognize arbitrary GUI elements from various desktop programs, web browsers, etc?
Re:R1 Computer Use by mountainriver on Thursday February 06, 2025 @05:01PM (#42966940)
Qwen2.5-vl seems to be the best right now by our tests.

UI-TARS by bytedance also has a good amount of pretraining.

Molmo is also very good at coordinates.