colinator

I concur. Most of my MCP pain is dealing with client's differing ability to handle images. Some clients (old codex) would even truncate the base64 data regardless of how it was json-wrapped. And sometimes they just ingest the base64 date directly into their context window. Not sure if this is an MCP thing or a clients-poorly-implementing-MCP thing.

Good move getting a nice robot! I'm doing something similar to you, but I went with a cheap robot, the "HIWONDER 6DOF Robotic Arm Kit". It was only $600, but wow is it bad. The precision and repeatability are both "are you drunk?" level. I can hear the gears grind when it moves. I suppose I should upgrade. But if my system can work with a terrible robot, I assume it would work even better with a nice one!

My project is https://github.com/colinator/Ariel - basically, no VLAs - instead, "just write code". Or have the agents do it.

I don't have a writeup yet about applying Ariel to _this_ robot, but this is for a previous one: https://colinator.github.io/Ariel/post1.html.

Excited to follow your progress!

That video is pretty good, thanks for finding it. I'm basically betting that an earlier, abandoned approach described in the video, "Code as Policy", will beat everything else. It requires no training data, and generalizes instantly to all robots.

I was also just in the market for a small experiment robot. I got the hiwonder armpi-fpv. Avoid it, the actuators are pretty bad - they're very 'grindy', the robot jitters like crazy when it moves. Any such problems with the lekivi?

For the fine-motor commands: or, the model can write the code to generate them on the fly. It seems to work, in my very limited experiment.

As for memory: my approach is to give the robot a python repl and, basically, a file system - the LLM can write modules, poke at the robot via interactive python, etc.

Basically, the LLM becomes a robot programmer, writing code in real-time.

This seems perfect to hook up to my 'LLMs can control robots over MCP' system. The idea is that LLMs are great at writing code, so let's lean in to that. I'll give it a try! I just got a bigger robot, we'll see how it does...

https://colinator.github.io/Ariel/post1.html

Remember how all those years ago (oh wait, a few months ago), that big npm hack attack was found by someone thinking their ssh connection was half a second too slow? Hope this ain't like that!

But seriously, good luck!

Yeah, the mechanism by which the LLMs control the robot is by writing code. I suppose they could also issue direct joint sequences, but I thought that they're so good at writing code already, might as well do that. So if they 'wanted' to they could write code with an explicit joint sequence they calculate in-context. That one seems more difficult.

So they can go 'slow', by taking a camera image, controlling the robot, repeating. Or they can write code that runs closer to the robot in a loop, either way. I thought the latter was somehow more impressive, and that's what you see in the hand-tracking example.

I'm connecting LLMs directly to robots, to see how well they can perform robot things by directly controlling motors and sampling the camera/sensors. Initial results are encouraging!

https://colinator.github.io/Ariel/post1.html

I just got a bigger robot, further results forthcoming!

Here's an idea for how to do that: treat frontier AI as a sort of 'common carrier'. The only business that frontier AI labs are allowed to conduct is selling raw tokens - no UI. Thus, 'claude code' would have to come from some other company. This would segment the AI industry, and, maybe, prevent a single entity (or small number of entities) from capturing all value.

Just a thought, what do you think?

Yet another problem with MCP: every LLM harness that does support it at all supports it poorly and with bugs.

The MCP spec allows MCP servers to send back images to clients (base64-encoded, some json schema). However:

1) codex truncates MCP responses, so it will never receive images at all. This bug has been in existence forever.

2) Claude Code CLI will not pass those resulting images through its multi-modal visual understanding. Indeed, it will create an entirely false hallucination if asked to describe said images.

3) No LLM harness can deal with you bouncing your local MCP server. All require you to restart the harness. None allow reconnection to the MCP server.

I assure you there are many other similar bugs, whose presence makes me think that the LLM companies really don't like MCP, and are bugly-deprecating it.

I've got one of these! Mine is called 'roboflex' (github.com/flexrobotics). It's c++/python, not rust. But similarly born out of frustration with ros. Writing your own robotics middleware seems to be a rite of passage. Just like 'writing your own game engine'. Nothing wrong with that - ros is powerful but has legit problems, and we need alternatives.

Although tbh, these days I'm questioning the utility. If I'm the one writing the robot code, then I care a lot about the ergonomics of the libraries or frameworks. But if LLMs are writing it, do I really care? That's a genuine, not rhetorical question. I suppose ergonomics still matter (and maybe matter even more) if I'm the one that has to check all the LLM code....

A few years ago there were actually two companies trying to manufacture "zblan optical fiber" (which has better light transmission than normal optical fiber) in orbit: Made In Space, and FOMS. Both of their websites are tombstones now, afaik. The former was also attempting 3d printing in space, and was bought by Redwire.

Fascinating tech, but seemed to go nowhere.

There are now several 'manufacturing in space platform' companies, like Varda. It's not enough to just be a platform. There needs to be an actual killer app.

Borges: Selected Non-fictions. Think his fictions are good? His non-fictions, imho, are even better. You can read three sentences and feel like you just listened to a symphony - you get that constant Borges wit, erudition, mystery. The English translations are SO good. Are they even better in Spanish?

I did some robotics tactile research, it was super fun! We used "biotac" sensors, which are very capable, but are 1) crazy expensive and 2) crazy hard to replace the skins, which do wear out.

One advantage biotacs have over these is that I can send a guy a (very large) check and buy them. Most academically-sourced things like this cannot be gotten for any price. These look cool, I'd love to have a few.

Here's another one: "roboflex", https://github.com/flexrobotics

ROS is irredeemable. I've seen it used in large projects, and the amount of time wasted "wrestling with ros" was ridiculous. I believe this will never change. Just because tensorflow existed meant folks shouldn't have looked for an easier way, and created PyTorch? No! Forge on, intrepid ros-replacers!

Well, our brains are closer to spiking neural networks than 'regular' neural networks. And they work pretty well. For the most part.

I feel like SNNs are like Brazil - they are the future, and shall remain so. I think more basic research is needed for them to mature. AFAIK the current SOTA is to train them with 'surrogate gradients', which shoe-horn them into the current NN training paradigm, and that sort of discards some of their worth. Have biologically-inspired learning rules, like STDP, _really_ been exhausted?

Robotics middleware: https://github.com/flexrobotics/roboflex

Basically a simpler version of ros. Easily, and performantly (is that a word?) connect cameras and other sensors, via xtensor/eigen/numpy to whatever algorithms you have, and control actuators/robots.

Oh yeah, "politely waiting". You mean "denying the very existence of a problem". Meanwhile: is it? How do you know?

I'm curious about this as well. It's now #69, 5 minutes later. I seem to recall this happening quite a bit for global warming stories. Anyone from HN want to explain please? Do all 'political' stories get buried?

What if there were a border-adjustment carbon tax? So countries that emit a lot of CO2 get taxed into oblivion? That's a thing we could do... (_will_ do: another story)

Yeah, totally! You can find me email from my github profile.

Thanks! No, not quite there yet. Most other libraries are much further along - mine is really an 'exercise'...

This neuromorphic stuff is super cool. I wish I could just buy some! I'm the author of spikeflow (github.com/colinator/spikeflow). I'd love to support neuromorphic hardware.

For the moment, biological neural networks (brains) are, I believe, many orders of magnitude more powerful, and especially more efficient than, current computers. But it's hard to say, because it's so hard to tease out the important parts of what the brain does, because it does so much. There are so many types of neurons, so many types of connectivity, so many interacting neurotransmitters, so many support cells that affect computation... almost like the brain is the result of a billion years of hack-upon-hack evolution...

Really? Well then, at least a handful of folks here would be on the buy side. Re sell side: what would it take to sell your expertise on such a service? I've done freelance work in another programming field, and for very short things like this (an hour or two here or there), I'd envision wanting to charge quite a bit. I dunno. Feel free to dm me - email's in profile.

Huh, I didn't know of that - is that UK only? But from a quick perusal - yeah, exactly like that!

I use an iPad app called Stretchpaper. Basically an infinite-zoom drawing surface. Disclosure I'm the dev. There's no Mac app though. http://stretchpaper.com