News for hackers. Stuff that matters.
Re:Meta Releases LLM Compiler by chad1n on Friday June 28, 2024 @01:29PM (#40822851)
As usual, Twitter is impressed by this, but I'm very skeptical, the chance of it breaking your program is pretty high. The thing that makes optimizations so hard to make is that they have to match the behavior without optimizations (unless you have UBs), which is something that LLMs probably will struggle with since they can't exactly understand the code and execution tree.
Re:Meta Releases LLM Compiler by cec on Friday June 28, 2024 @04:17PM (#40824813)
Hey! The idea isn't to replace the compiler with an LLM, the tech is not there yet. Where we see value is in using these models to guide an existing compiler. E.g. orchestrating optimization passes. That way the LLM won't break your code, nor will the compiler (to the extent that your compiler is free from bugs, which can tricky to detect - cf Sec 3.1 of our paper).
Re:Meta Releases LLM Compiler by swyx on Friday June 28, 2024 @08:03PM (#40826694)
then maybe dont name it "LLM Compiler", just "Compiler Guidance with LLMs" or "LLM-aided Compiler optimization" or something - will get much more to the point without overpromising
Re:Meta Releases LLM Compiler by nickpsecurity on Friday June 28, 2024 @11:37PM (#40827714)
Yeah, the name was misleading. I thought it was going to be source to object translation maybe with techniques like how they translate foreign languages.
Re:Meta Releases LLM Compiler by verditelabs on Friday June 28, 2024 @05:07PM (#40825317)
I've done some similar LLM compiler work, obviously not on Meta's scale, teaching an LLM to do optimization by feeding an encoder/decoder pairs of -O0 and -O3 code and even on my small scale I managed to get the LLM to spit out the correct optimization every once and a while.

I think there's a lot of value in LLM compilers to specifically be used for superoptimization where you can generate many possible optimizations, verify the correctness, and pick the most optimal one. I'm excited to see where y'all go with this.

Re:Meta Releases LLM Compiler by viraptor on Friday June 28, 2024 @05:25PM (#40825442)
Thank you for freeing me from one of my to-do projects. I wanted to do a similar autoencoder with optimisations. Did you write about it anywhere? I'd love to read the details.
Re:Meta Releases LLM Compiler by verditelabs on Friday June 28, 2024 @05:34PM (#40825516)
No writeup, but the code is here:

https://github.com/SuperOptimizer/supercompiler

There's code there to generate unoptimized / optimized pairs via C generators like yarpgen and csmith, then compile, train, inference, and disassemble the results

Re:Meta Releases LLM Compiler by hughleat on Saturday June 29, 2024 @11:50AM (#40831425)
Yes! An AI building a compiler by learning from a super-optimiser is something I have wanted to do for a while now :-)
Re:Meta Releases LLM Compiler by moffkalast on Friday June 28, 2024 @06:02PM (#40825744)
> The idea isn't to replace the compiler with an LLM, the tech is not there yet

What do you mean the tech isn't there yet, why would it ever even go into that direction? I mean we do those kinds of things for shits and giggles but for any practical use? I mean come on. From fast and reliable to glacial and not even working a quarter of the time.

I guess maybe if all compiler designers die in a freak accident and there's literally nobody to replace them, then we'll have to resort to that after the existing versions break.

Re:Meta Releases LLM Compiler by Lockal on Friday June 28, 2024 @04:07PM (#40824705)
As this LLM operates on LLVM intermediate representation language, the result can be fed into https://alive2.llvm.org/ce/ and formally verified. For those who don't know what to print there: here is an example of C++ spaceship operator: https://alive2.llvm.org/ce/z/YJPr84 (try to replace -1 with -2 there to break). This is kind of a Swiss knife for LLVM developers, they often start optimizations with this tool.

What they missed is to mention verification (they probably don't know about alive2) and comparison with other compilers. It is very likely that LLM Compiler "learned" from GCC and with huge computational effort simply generates what GCC can do out of the box.

Re:Meta Releases LLM Compiler by boomanaiden154 on Friday June 28, 2024 @04:40PM (#40825046)
I'm reasonably certain the authors are aware of alive2.

The problem with using alive2 to verify LLM based compilation is that alive2 isn't really designed for that. It's an amazing tool for catching correctness issues in LLVM, but it's expensive to run and will time out reasonably often, especially on cases involving floating point. It's explicitly designed to minimize the rate of false-positive correctness issues to serve the primary purpose of alerting compiler developers to correctness issues that need to be fixed.

Re:Meta Releases LLM Compiler by hughleat on Friday June 28, 2024 @06:00PM (#40825719)
Yep, we tried it :-) These were exactly the problems we had with it.
Re:Meta Releases LLM Compiler by boomanaiden154 on Friday June 28, 2024 @04:53PM (#40825182)
I'm not sure it's likely that the LLM here learned from gcc. The size optimization work here is focused on learning phase orderings for LLVM passes/the LLVM pipeline, which wouldn't be at all applicable to gcc.

Additionally, they train approximately half on assembly and half on LLVM-IR. They don't talk much about how they generate the dataset other than that they generated it from the CodeLlama dataset, but I would guess they compile as much code as they can into LLVM-IR and then just lower that into assembly, leaving gcc out of the loop completely for the vast majority of the compiler specific training.

Re:Meta Releases LLM Compiler by hughleat on Friday June 28, 2024 @06:01PM (#40825731)
Yep! No GCC on this one. And yep, that's not far off how the pretraining data was gathered - but with random optimisations to give it a bit of variety.
Re:Meta Releases LLM Compiler by boomanaiden154 on Friday June 28, 2024 @06:38PM (#40826071)
Do you have more information on how the dataset was constructed?

It seems like somehow build systems were invoked given the different targets present in the final version?

Was it mostly C/C++ (if so, how did you resolve missing includes/build flags), or something else?

Re:Meta Releases LLM Compiler by hughleat on Saturday June 29, 2024 @11:55AM (#40831471)
We plan to have a peer reviewed version of the paper where we will probably have more details on that. Otherwise we can't give anymore details than in the paper or post, etc. without going through legal which takes ages. Science is getting harder to do :-(
Re:Meta Releases LLM Compiler by moffkalast on Friday June 28, 2024 @06:17PM (#40825887)
> C++ spaceship operator

> (A <=> B) < 0 is true if A < B

> (A <=> B) > 0 is true if A > B

> (A <=> B) == 0 is true if A and B are equal/equivalent.

TIL of the spaceship operator. Was this added as an april fools?

Re:Meta Releases LLM Compiler by samatman on Friday June 28, 2024 @06:50PM (#40826170)
This is one of the oldest computer operators in the game: the arithmetic IF statement from FORTRAN.

It's useful for stable-sorting collections with a single test. Also, overloading <=> for a type, gives all comparison operators "for free": ==, !=, <, <=, >=, >

Re:Meta Releases LLM Compiler by pklausler on Friday June 28, 2024 @09:51PM (#40827289)
It also has no good definition for its semantics when presented with a NaN.
Re:Meta Releases LLM Compiler by sagarm on Saturday June 29, 2024 @03:23PM (#40832810)
Re:Meta Releases LLM Compiler by pklausler on Saturday June 29, 2024 @06:28PM (#40833705)
How would that apply to Fortran’s arithmetic IF statement? It goes to one label for a negative value, or to a second label for a zero, or to a third label for positive. A NaN is in none of these categories.
Re:Meta Releases LLM Compiler by moffkalast on Saturday June 29, 2024 @06:31AM (#40829212)
I mean maybe I'm missing something but it seems like it behaves exactly the same way as subtraction? At least for integers it's definitely the same, for floats I imagine it might handle equals better?
Re:Meta Releases LLM Compiler by samatman on Saturday June 29, 2024 @01:43PM (#40832251)
C++ has operator overloading, so you can define the spaceship for any class, and get every comparison operator from the fallback definitions, which use `<=>` in some obvious ways.
Re:Meta Releases LLM Compiler by sagarm on Saturday June 29, 2024 @03:21PM (#40832800)
The three-way comparison operator just needs to return a ternary, and many comparisons boil down to integer subtraction. strcmp is also defined this way.

In C++20 the compiler will automatically use the spaceship operator to implement other comparisons if it is available, so it's a significant convenience.

Re:Meta Releases LLM Compiler by solarexplorer on Friday June 28, 2024 @04:00PM (#40824641)
If I understand correctly, the AI is only choosing the optimization passes and their relative order. Each individual optimization step would still be designed and verified manually, and maybe even proven to be correct mathematically.
Re:Meta Releases LLM Compiler by boomanaiden154 on Friday June 28, 2024 @04:32PM (#40824971)
Right, it's only solving phase ordering.

In practice though, correctness even over ordering of hand-written passes is difficult. Within the paper they describe a methodology to evaluate phase orderings against a small test set as a smoke test for correctness (PassListEval) and observe that ~10% of the phase orderings result in assertion failures/compiler crashes/correctness issues.

You will end up with a lot more correctness issues adjusting phase orderings like this than you would using one of the more battle-tested default optimization pipelines.

Correctness in a production compiler is a pretty hard problem.

Re:Meta Releases LLM Compiler by hughleat on Friday June 28, 2024 @06:06PM (#40825786)
There are two models.

- foundation model is pretrained on asm and ir. Then it is trained to emulate the compiler (ir + passes -> ir or asm)

- ftd model is fine tuned for solving phase ordering and disassembling

FTD is there to demo capabilities. We hope people will fine tune for other optimisations. It will be much, much cheaper than starting from scratch.

Yep, correctness in compilers is a pain. Auto-tuning is a very easy way to break a compiler.

Re:Meta Releases LLM Compiler by ramon156 on Friday June 28, 2024 @01:45PM (#40823033)
Let's be real, at least 40% of those comments are bots
Re:Meta Releases LLM Compiler by extheat on Friday June 28, 2024 @02:48PM (#40823783)
People simply have no idea what they're talking about. It's just jumping on to the latest hype train. My first impression here was per the name that it was actually some sort of compiler in it of itself--ie programming language in and pure machine code or some other IR out. It's got bits and pieces of that here and there but that's not what it really is at all. It's more of a predictive engine for an optimizer and not a very generalized one for that.

What would be more interesting is training a large model on pure (code, assembly) pairs like a normal translation task. Presumably a very generalized model would be good at even doing the inverse: given some assembly, write code that will produce the given assembly. Unlike human language there is a finite set of possible correct answers here and you have the convenience of being able to generate synthetic data for cheap. I think optimizations would arise as a natural side effect this way: if there's multiple trees of possible generations (like choosing between logits in an LLM) you could try different branches to see what's smaller in terms of byte code or faster in terms of execution.

Re:Meta Releases LLM Compiler by quonn on Friday June 28, 2024 @03:55PM (#40824599)
> Presumably a very generalized model would be good at even doing the inverse: given some assembly, write code that will produce the given assembly.

ChatGPT does this, unreliably.

Re:Meta Releases LLM Compiler by hughleat on Friday June 28, 2024 @06:09PM (#40825828)
It can emulate the compiler (IR + passes -> IR or ASM).

> What would be more interesting is training a large model on pure (code, assembly) pairs like a normal translation task.

It is that.

> Presumably a very generalized model would be good at even doing the inverse: given some assembly, write code that will produce the given assembly.

Is has been trained to disassemble. It is much, much better than other models at that.

Re:Meta Releases LLM Compiler by bbor on Friday June 28, 2024 @04:10PM (#40824738)
AFAIK this is a heuristic, not a category. The underlying grammar would be preserved.

Personally I thought we were way too close to perfect to make meaningful progress on compilation, but that’s probably just naïveté

Re:Meta Releases LLM Compiler by boomanaiden154 on Friday June 28, 2024 @04:34PM (#40824997)
I would not say we are anywhere close to perfect in compilation.

Even just looking at inlining for size, there are multiple recent studies showing ~10+% improvement (https://dl.acm.org/doi/abs/10.1145/3503222.3507744, https://arxiv.org/abs/2101.04808).

There is a massive amount of headroom, and even tiny bits still matter as ~0.5% gains on code size, or especially performance, can be huge.

Re:Meta Releases LLM Compiler by namaria on Saturday June 29, 2024 @04:16AM (#40828648)
This feels like going insane honestly. It's like reading that people are super excited about using bouncing castles to mix concrete.