Nice! Thanks for confirming. If the last instruction is 'go back to where you started' does it work?

Yes, it would work.

Please oh please try to make Kärcher adopt your stuff. Even their latest premium cleaning bots are hopeless when they don't know where they are, even when I tell them where they are.

Or, I don't know, make your own vacuums.

I have no hope for this working. I recently bought a dumb meatbag operated vacuum after tiring of the robot's issues. The constant getting stuck in places drove me crazy. After getting a new vac, I used it in a place where the robot had just "cleaned" that morning. According to the new collector's contents, the robot sucked at its job of sucking.

The sad part is that my previous robot (iRobot model from 3 years ago) was better at this.

Mine worked pretty well as well and slightly improved after an update, but has gotten dumber over the years. Not only does it get lost, but it physically gets stuck under things that it did not used to get stuck under.

what kind of lunatic allows a camera-equipped LLM into their house and lets it drive itself around?

This is Mistral though. I assume the model can be run locally.

So the vacuum cleaner manufacturer would just require customers to install mistral on their local LLM hostbox? Or maybe they'd sell you a host to go with it you couldn't get into and honest cross fingers we aren't peeking!

It is an 8B model… so, I guess it probably couldn’t run directly on most existing vacuum robots, but it could be reasonable this decade, right?

I wish we lived in the alternate universe where everybody has a home server that could run this sort of thing.

Once prices settle down it doesn’t seem unreasonable that everyone who wants modern conveniences plops down $3k for a home computer. It’s still cheaper than during the “home computer revolution” 80s/90s when normies were buying these at Sears and Radio Shack.

It would be nice to believe that could be possible, but people mostly switched to cheap laptops as soon as they became viable. No harm in being hopeful though.

That would be a rather expensive robot vacuum.

True. But it would probably take a couple years to get spin up a new hardware product line anyway; by then local inference will probably be cheaper, right?

Can you please stop posting flamebait, unsubstantive comments, name-calling, shallow dismissals and the like? You've been doing these things repeatedly. It's not what this site is for, and destroys what it is for.

We've already asked you this a couple times. If you'd please review the site guidelines at https://news.ycombinator.com/newsguidelines.html and follow them, we'd appreciate it.

You got it hoss

Are you interested in working with partners that are collecting training data?

So, it has to rely on exact situational step by step commands? I'm wondering how one could conceivably deploy this in a useful way. Usually you'd need to mark areas on the map and then the robot knows where to go, A* is trivial around obstacles once you have that and a lidar. And lidars are an order of magnitude cheaper than something that can run an 8B VLA.

One could maybe autogenerate these text planning commands, but it would require a map and the robot's current location, so it doesn't really solve that, unless it can find a specific thing completely on its own. How much of a planning horizon does it have?

You probably don't need a geometric map. Just have someone wander around with a mobile app and feed the video into a more powerful model once, asking it to produce descriptions of the different areas of the office or building and how they connect. Now you have a "text adventure game" map you can use it with a small LLM to produce instructions for the robot to follow, assuming it knows where it currently is.

The advantage over traditional approaches is presumably flexibility. LIDAR isn't going to solve an instruction like "find the man with the pink shirt".

Yeah the benefits of a model that can understand text well enough is certainly that one could in theory just describe the layout. Just let it know what connects to what, what's on each floor, etc. Might work if it can figure out where it is based on contextual info, but probably not super well yet I imagine. VLAs are known to have very little text context, like sentences worth at most, maybe this one's different.

Could you please open source this or a 4B version? I’ve been messing around w hooking up vllms to cheap robots and skipping the whole ROS stack and this would be an absolute delight to play with

Do you think a similar model architecture could include some sort of memory, so a follow-up prompt of "Go back to the water foutain" would somehow work?

A in-model memory approach is probably still deep research but maybe a Rag-like pipeline could work in some instances