I do wonder if auto-research would have reached something similar, it did take a significant amount of steering from me to get it to the point where it was working realtime.

For anyone interested the ffmpeg is at https://github.com/poizan42/jellyfin-rpi-ffmpeg and a shim for using it with stock jellyfin at https://github.com/poizan42/jellyfin-rpi-ffmpeg-shim

I then asked claude to compare the C#/.NET implementation in the library with the C++ version, and it quickly identified that the C# library was missing a couple of fairly cheap optimisations that were present in the C++ version.

If I can help get a PR merged, then it'll be by far the biggest impact of any work I've ever done.

I also compared the Rust version, it had this specific optimisation. The far more popular Tokio/Prost library did not.

Given appropriate guardrails, LLMs are impossibly fast at iterating to find root causes and specific performance bottlenecks.

Could you have manually profiled and compared the execution paths? Sure. Could you have translated the C++ optimizations to C#? Sure. But in such an obvious case, the LLM managed itself.

I’ve been using fable recently not to write code, but to write briefing notes on projects. “Read and compare these three libraries. What are the unique differences in their implementations? Why were those ideas chosen? What are the pros and cons of those designs? What ideas would you copy into a new project? Write a report on your findings.”

Doing the same, re-implementing a lot of LLM/diffusion models in Rust+CUDA for my own usage, usually the initial implementation takes 1-2 days (of 100% autonomous work) then I put an agent to optimize the implementation which tends to get close to SOTA performance within another day or two.

As long as you can point the agent at "This is the correct baseline, make sure any optimizations still pass this", seemingly you can leave them and they come back after N hours with a faster program that just works.

https://github.com/qarl/arcade-js

In Time Pilot - there are three routines that are called constantly from inside the main loop. Each routine computes the checksum of the other routine's code to see if it's been modified. If so it jumps into random junk data.

There are other less exotic routines that make sure the copyright string hasn't been modified, etc.

https://github.com/qarl/arcade-js/blob/main/games/timeplt/id...

Fascinating.

Even the cheap LLMs are great in doing the awful crud work in the beginning: finding offsets, firmware update file structures, brute forcing checksums, etc.

It still produces a lot of crap in the later steps (understanding the implementation itself) but I'm happy doing this stuff myself then.

I've had success here by adding a phase called "grounding" that attempts to verify its "understanding" by creating tests that modify the running executable to ensure its made the right inference.

Is this variable really MARIO_X? Change it and see if Mario moves. Etc.

As an example in Donkey Kong - the system had trouble deciding if an array controlled barrels or fireballs. There was conflicting evidence.

After many trips through the loop - it realized it does BOTH, depending on which level you're on.

So the "understanding" grows with each iteration.

with opus 5, I'm finding that asking itself to write such a loop skill that does benchmark - profile - verify - research - improve, keep logs for each iteration, etc. (just like autoresearch), and then using the skill, works very nicely.

Watching claude and codex play winquake and age of empires, and debug support for Firefox 52 has been wild.

This!

People keep measuring how good AI is by one-shotting a problem, but I e found that a back and fourth via the Superpowers SKILL.md is how the model builds these constraints… then when it has all the holes, it gets to work