News for hackers. Stuff that matters.

A comment on R1 Computer Use.

Re:R1 Computer Use by emregucerr on Thursday February 06, 2025 @04:39PM (#42966766)
i wonder how good is R1 at counting pixels from a screenshot. what enabled claude and OAI's CUA to develop computer use was being able to precisely give x-y coordinates of a click location.

also, how big of a gain to have reasoning for computer use? i feel like reasoning unlocks a lot when there is a single complex question but not so much better at taking actions in a long term plan.

[ Parent | on HN ]
Re:R1 Computer Use by mountainriver on Thursday February 06, 2025 @05:00PM (#42966928)
Yep, coordinate grounding is key, we use Ai2's pixmo for a lot of that https://huggingface.co/datasets/allenai/pixmo-points

We had previously created https://huggingface.co/datasets/agentsea/wave-ui but that was superseded by pixmo as it contains over a million data points.

[ Parent | on HN ]