rjpower9000

“They want that land to be clean corn and soybeans,” Bishop said. Before the restrictions, his father was growing organic corn and soybeans on part of the field and letting Bishop grow vegetables on the rest.

I've seen this mentioned elsewhere, but the idea that you'd force someone else to create a mono-crop desert, not even out of a sense of efficiency, but _just because it looks right_, is just so frustrating.

It's hard to imagine how a company could be more extractive than this.

Here you've got the advantage that you're repeating the same task over and over, so you can tweak your prompt as you go, and you've got the "spec" in the form of the C code there, so I think there's less to go wrong. It still did break things sometimes, but the fuzzing often caught it.

It does require careful prompting. In my first attempt Claude decided that some fields in the middle of an FFI struct weren't necessary. You can imagine the joy of trying to debug how a random pointer was changing to null after calling into a Rust routine that didn't even touch it. It was around then I knew the naive approach wasn't going to work.

The second attempt thus had a whole bunch of "port the whole struct or else" in the prompt: https://github.com/rjpower/zopfli/blob/master/port/RUST_PORT... .

In general I've found the agents to be a mixed bag, but overall positive if I use them in the right way. I find it works best for me if I used the agent as a sounding board to write down what I want to do anyway. I then have it write some tests for what should happen, and then I see how far it can go. If it's not doing something useful, I abort and just write things myself.

It does change your development flow a bit for sure. For instance, it's so much more important to concrete test cases to force the agent to get it right; as you mention, otherwise it's easy for it do something subtly broken.

For instance, I switched to tree-sitter from the clang API to do symbol parsing, and Claude wrote effectively all of it; in this case it was certainly much faster than writing it myself, even if I needed to poke it once or twice. This is sort of a perfect task for it though: I roughly knew what symbols should come out and in what order, so it was easy to validate the LLM was going in the right direction.

I've certainly had them go the other way, reporting back that "I removed all of the failing parts of the test, and thus the tests are passing, boss" more times than I'd like. I suspect the constrained environment again helped here, there's less wiggle room for the LLM to misinterpret the situation.

You could certainly try using c2rust to do the initial translation, and it's a reasonable idea, but I didn't find the LLMs really struggled with this part of the task, and there's certainly more flexibility this way. c2rust seemed to choke on some simple functions as well, so I didn't pursue it further.

And of course for external symbols, you're constrained by the C API, so how much leeway you have depends on the project.

You can also imagine having the LLM produce more idiomatic code from the beginning, but that can be hard to square with the incremental symbol-by-symbol translation.

That said, you could have the LLM write equivalence tests, and you'd still have the top-level fuzz tests for validation.

So I wouldn't say it's impossible, just a bit harder to mechanize directly.

I didn't measure consistently, but I would guess 60-70% of the symbols ported easily, with either one-shot or trivial edits, 20% Gemini managed to get there but ended up using most of its attempts, and 10% it just struggled with.

The 20% would be good candidates for multiple generations & certainly consumed more than 20% of the porting time.

I guess I worry it would be hard to separate out the "noise", e.g. the C code touches some memory on each call so now the Rust version has to as well.

Indeed, this is exactly the type of subtle case you'd worry about when porting. Fuzzing would be unlikely to discover a bug that only occurs on giant inputs or needs a special configuration of lists.

In practice I think it works out okay because most of the time the LLM has written correct code, and when it doesn't it's introduced a dumb bug that's quickly fixed.

Of course, if the LLM introduces subtle bugs, that's even harder to deal with...

https://lwn.net/Articles/820424/

The objections are of course reasonable, but I kept thinking this shouldn't be as big a problem in the future. A lot of times we want to make some changes that aren't _quite_ mechanical, and if they hit a large part of the code base, it's hard to justify. But if we're able to defer these types of cleanups to LLMs, it seems like this could change.

I don't want a world with no API stability of course, and you still have to design for compatibility windows, but it seems like we should be able to do better in the future. (More so in mono-repos, where you can hit everything at once).

Exactly as you write, the idea with prompts is that they're directly actionable. If I want to make a change to API X, I can test the prompt against some projects to validate agents handle it well, even doing direct prompt optimization, and then sharing it with end users.

I came to the conclusion that while I didn't necessarily _like it_ per se, I had to acknowledge how absurdly talented Joyce was, and that there was some justification for being in the top books list. My feeling was that the lack of enjoyment was a fault of the book but more that I didn't have the background to appreciate it. Though there were also some chapters where most people agree Joyce was just trying too hard and it shows.

> It's just that all three steps are really really hard.

Haha, I was hoping for something a little easier than that! I'm not sure gradient descent would apply for all problem spaces, but I get the gist of what you're saying.

> TDDs insight is that gradient descent is relatively easy if the utility function is one-dimensional and monotonic. (Bonus point, it still works with the set of initial ideas being empty)

That's sort of what I was driving at. I certainly won't argue you can't, with time and patience, at least exhaustively enumerate a solution space.

My impression of the TDD literature e.g. things like https://en.wikipedia.org/wiki/Transformation_Priority_Premis... is that they're pushing an idea that you can systematically walk through a set of transformations and get a program, that we can thus avoid the "really really hard" steps you mention.

This hill-climbing style matches closely with the monotonic utility function you mention. And if there are lots of interesting problems where this works for people, then that's great. I certainly won't object to having a system for approaching problems, and the general idea of trying to avoid adding complexity too early.

My original motivation was really just observing what appeared to happen when you apply these techniques _outside of their scope_. The failure mode becomes this sort of fascinating circling around a local minimum, with local changes that don't really make progress towards the ultimate goal. This is exactly what you'd see in an ML domain so it's kind of interesting to see it in the real-world.

Mea culpa: that was the part I found really interesting. I likely tried to stretch the point too broadly. Ultimately what I wanted to convey was there's no general way to avoid the hard part.

My point was more about arbitrary programs and tasks. In general, I don't think there's any particular process you can follow that, in effect, reduces programming or math to a checklist. I'm sure there are techniques that can help structure many problems but they're necessarily fairly general e.g. "think about the problem, take a nap, think about it some more".

What I was hoping to highlight was more the danger of trying to hill-climb without having the right set of skills, thus assuming you can solve a problem by following a particular pattern.

To give an example, I remember trying to derive the quadratic equation when I was young, and no amount of algebraic repositioning was going to help me unless I know how to complete the square! But once that type of trick is in your toolbox, you can begin to solve all sorts of other problems.

Even the bowling calculator can become hard if you don't know what your doing, not because TDD is bad or a wrong way to do things, but because you don't know how to reason about your program. Even bowling has 12^13 outcomes, with a bunch of silly special cases if you approach it the wrong way. Thus a truly naive approach might lead you down a weird direction.

For example with the bowling score calculator, it's great to start with some tests, but I think he then marched towards a specific OOP formulation which obscured rather than clarified the problem.

Good point. More broadly than just TDD, we might characterize an "easy" problem as one where the surface from conception to completion is broadly visible and there's a logical set of steps (i.e. "differentiable"). Or maybe "easy" is I can see the full set of steps immediately, and "tractable" is that I've got a good guess.

But the set of steps you can take is highly dependent on your knowledge and skillset. For example, if I know how about completing the square, then deriving the quadratic equation is a basic algebra exercise. If I don't, then I might struggle for a long time before I manage to discover that key insight.

I'm sure there are coding methodologies which can lead to better or worse results, but they augment knowledge or skill, they don't replace it.

Sometimes it does it right, some times not. I can see the relation to gambling if, say, it does it right 50% of the time. If I had been taking a more scientific approach to the problem and had a clear direction of what I wanted to test, I suspect I wouldn't have gotten quite as "stuck".

That's a better description than what I came up with, "Tiktok for engineers". LLMs probably compound the issue, with that hope of a magic outcome. Though I've had many problems pre-LLM where I was plowing through it myself and couldn't let up...

I doubt that's the case. (The yard waste and recycling bins are more full in my neighbourhood than the garbage cans...)

I can't speak for other languages, but for C++, it's mainly the huge build time just to get started using the project. This is obviously an issue for any C++ project, but with GRPC, having to build all of OpenSSL is particularly annoying. (Especially if you're not using secured channels!).

I also couldn't figure out how to build just the C++ static libraries easily. Instead I end up building everything and then deleting the shared libraries. That's easier to do than trying to figure out how to convince the linker that I prefer to link statically.

Once that's out of the way it's pretty reasonable. It took a while to figure out that `--cpp_out` doesn't actually generate stubs, though I'm sure it's documented.

So it's not really worse than any other package; it's just _big_, and that makes it a little more annoying. Having any kind of reasonable package manager for C would simplify things dramatically. Not having to express my dependency on GRPC as a git submodule+some clunky Makefile rules would be nice.

The thing is it's really easy for the Fed to slow the economy down, it's a lot harder to get it started again. So it makes sense for them to wait until they actually start seeing inflation before they decide to raise rates.

It's hard to say whether the mal-investment is due to the current monetary policy or the lack of better investment options in a bad economy.

FWIW, the participation rate for the wider population shows a more significant drop around 2009:

https://research.stlouisfed.org/fred2/series/CIVPART

This might be due to early retirement, or more young people not finding work. I don't research this data much.

I've been toying with a similar idea that would focus only on static libraries and would handle binaries as an optimization. Just being able to specify "fetch zlib, protocol buffers and libjpeg" and build them for me would simplify many of my C projects.

The only coherent way I could think of to manage architectural differences was to force a single build tool that understood the important compiler flags and version incompatibilities.

Building from source solves a lot of problems but somehow even today it still takes forever to compile certain projects (looking at you, grpc).