mplappert

Even back in 2021, OpenAI worked on learned optimizers [1], ie neural networks that can update neural networks. Looks like this is now part of their larger model runs …

[1]: https://arxiv.org/abs/2106.00958

If you watch the live stream for a bit it’s also obvious that the harness makes it very cumbersome for them to do anything. They spend lots of time just coordinating hand-offs and seem to have delayed inventory access etc.

This is exactly and 100% my experience with these agents: They drown themselves in process and insane levels of bureaucracy surprisingly quickly.

A couple months ago we had an experiment where we had an agent run a small business [1] for other agents and exactly the same happened: Crazy ledgers, endless analysis, and way too much attention to inconsequential decisions.

At one point our agent wrote elaborate date linters because it had a minor time mix-up.

[1]: https://dfdxlabs.com/research/2026/hans-kraemer

I agree except that it usually is correct code (at least in the sense of it passing some unit tests) because you also RL directly against that.

But yes, in my experience the code is often quite bad in other, harder-to-quantify directions: readability, maintainability, simplicity.

> After that result, the model added the researcher’s locally available GitHub token to the retrieval program. The recorded command calls `gh auth token` and writes the token as separate string literals. Its recorded reasoning stated an aim of avoiding secret scanning, the automated checks for exposed tokens. The new commit and push succeeded.

Not sure exactly how that works since I’d imagine that the key needs to be there in full eventually?

Apparently the agent had access to an OpenAI researcher’s own account, which in turn had _write_ access to the official OpenAI Codex repository. The agent then leaked a GitHub key to that repo.

But what’s crazy to me is that the agent had write access to the codex repo in the first place. WTF

“Never attribute to malice that which is adequately explained by stupidity.” (or carelessness in this case)

Seems like a hallucination to me; note that the context contains “unmarkBlock” as the function name, which invites a connection to Minecraft. Still shouldn’t happen of course.

The alternative explanation is that the inference engine, which batches several unrelated requests for parallel processing, messed up the unpacking and returned an unrelated user’s query. This one would be very scary as it will leak arbitrary content, but it seems much less likely here.

I agree with this. I recently went back to writing more code by hand again for this reason: the amount of effort / mental toll it takes me to steer the LLM is sometime just not worth it. I might be slower but I enjoy my work much more again.

There’s a lot of confusion here:

- There is no double taxation if you just pay yourself a salary (since it’s a normal business expense). If you want to take money out of the company flexibly, a GmbH is the wrong structure.

- I’ve never heard of anybody doing an UG/GmbH + KG to get started. This is highly unusual. Most people either do just a simple UG or maybe they set up a holding structure with two separate GmbH / UG entities.

- Related to the above: if you go with a simple, standard structure you will incur minimal legal fees. You don’t need a lawyer, you just directly task a notary and tell them you want a standard setup.

- If you don’t want the complexity of a limited liability company, the standard way to reduce liability risk is to get liability insurance. Many, many people do this instead of having a GmbH.

The valid criticism is the a) lack of digital processes and b) sequential processing of steps that could happen in parallel. For example, I sped up my own GmbH process by driving to the register court and paying in cash on-site. For whatever reason that’s much faster and saves about a week.

Yeah I tried to use ROS / Gazebo about 10 years ago (it was still ROS 1) and getting it set up was an immense pain. I remember that creating new modules required writing CMake files. Maybe this is better now, but I decided to skip that.

I also agree on the need for a simple, easy, extensible open source framework. LeRobot IMO is some of this but also contains the dataset + ML code. I think Flask is nice because it's so singularly focused on just one thing with extensibility if you need extras.

So I really like the idea. But being an OSS maintainer these days seems... intense.

I’d recommend looking at the SO-101 then. Much much cheaper.

And hardware is very fun! It’s also very frustrating. But to me worth it.

This all makes a lot sense, thanks for sharing!

For depth I agree on the VLA route but for ACT / DP-style imitation learning from scratch it seems more feasible (since you’re not fighting a pretrained model that was not trained on this modality). Might also increase robustness since you naturally end up with an input that’s invariant to colors / textures. Plan is to try both paths: the from scratch (and then ablate RGB vs RGB-D) and the VLA + fine-tuning one.

Did you ever a late RGB vs RGB-D? I haven’t seen that much in the literature.

Also any thoughts on action space representation? Seems to me people are settling on flow matching mostly, but pi still uses discrete tokens to supervise the upstream backbone VLM. I also like the simplicity of discrete bins and used that successfully in the past.

Yeah I can see that. Especially the rotation part is pretty awkward (hence why I currently do the axis parallel to table surface trick and only use the yaw angle to rotate the gripper). The space mouse was super easy to get started with though, so in terms of getting something up and running quickly it has an advantage.

This seems indeed useful. I haven't seen this for robotics but I'm sure people need this for larger deployments (either for a distributed fleet or for an "arm farm"-like setup where there's many robots in one location for data collection / eval). Interesting idea!

Wow very cool project and thanks for the Ariel pointer! Added to my reading list.

And yes, having a nice robot makes life so much easier. Very happy with my choice so far!

Oh very cool! Looks a bit like the TRLC-DK1 (I was looking at this one for a bit).

I think pushing the sota is quite hard to do solo but we'll see. Mostly I want to get back up to speed after having not done much robotics during the last 6 years. Best way for me to learn is to just do it, so here we are. We'll see how far I get (I suspect at some point compute will be the main bottleneck)

I timestamp everything twice: once with the hardware clock (if available, like for the realsense camera) and once within my robot stack once it gets read from the device (using `time.monotonic_ns()`). Both are stored and alignment can happen with either timestamp. I think the 2nd timestamp is actually more meaningful since ultimately I want to reconstruct the state that the policy would've seen; so if one modality is delayed I should actually include that effect during training.

That being said, I might switch to a realsense for the static tabletop camera as well; the realsense wrist is clearly much more reliable than the cheap Logitech C920 that I currently use.

I'm only starting down this road but my sense is that ACT and Diffusion Policy both make it pretty feasible to start on real data only. LeRobot also makes it easier to train these. But that's the next step that I'm working on, so I don't know yet.

On attachments: during this project I really wanted a 3D printer several times. So that's probably next on the shopping list.

The Ufactory arm is actually quite extensible: it exposes digital input/output and you have a standard wrist mount where you can mount different end effectors or attachments.

Cool video! What made you stop if I can ask?

And yeah I feel you re humanoid. I worked on the Rubik's cube project at OpenAI, which used a humanoid hand, and it was insanely painful and hard. Also fun anecdote: it was completely impossible to teleop the shadow hand. We had a data glove to capture hand movements but as soon as contact / haptics come in, you're lost. We could never even get a single rotation on the Rubik's cube via teleop.

I do think simpler hardware like the one described in my post works really though and it's so much easier to do something with it.

Very cool!

Have you seen the recent nvidia thing? They do this at scale for robotics manipulation: https://research.nvidia.com/labs/gear/enpire/

Great points and I very much appreciate the input!

Right now the static camera is probably really bad: It's mounted on my desk, so its very easy to bump into it and move it. So yeah, it's position for sure will change over time. I think I need a better solution, maybe a rail system that's more rigidly attached to the robot arm so that at least the camera stays fixed relative to that point of reference.

That was my take 8 years ago glad to hear it’s still that

For understanding: I think the level is much deeper if I wrote the code vs reading someone else’s. Same applies to coding agents of course which is why I wrote most of it myself and only delegate some tasks (for example codex was great help at setting up telemetry dashboards or writing the custom glfw renderer).

On control: LeRobot will change all the time and I’ll be unaware of what changed. If something suddenly doesn’t work anymore, it’s a pain to find out. I can of course fork or pin but that defeats the purpose a bit.

At the end it’s also partially just preference: I wanted to write this layer myself, and I have opinions about how it should be architected, so I did.