The code is of terrible quality and I am at 100+ comments on my latest PR.
That being said, my latest PR is my second-ever to LLVM and is an entire linter check. I am learning far more about compilers at a much faster pace than if I took the "normal route" of tiny bugfixes.
I also try to do review passes on my own code before asking for code review to show I care about quality.
LLMs increase review burden a ton but I would say it can be a fair tradeoff, because I'm learning quicker and can contribute at a level I otherwise couldn't.
edit: the PR in question. Unfortunately I've been on vacation and haven't touched it recently.
https://github.com/llvm/llvm-project/pull/146970
It's a community's decision whether to accept this tradeoff.
If unrepentant garbage that you make others mop up at risk of their own projects’ integrity is the level you aspire to, please stop coding forever.
If your project has a policy against AI usage I won't submit AI-generated code because I respect your decision.
That's not what the GP mean. Just because a community doesn't disallow something doesn't mean it's the right thing to do.
> I also try to mitigate the code review burden by doing as much review as possible on my end
That's great but...
> & flagging what I don't understand.
It's absurd to me that people should commit code they don't understand. That is the problem. Just because you are allowed to commit AI-generated/assisted code does not mean that you should commit code that you don't understand.
The overhead to others of committing code that you don't understand then ask someone to review is a lot higher than asking someone for directions first so you can understand the problem and code you write.
> If your project has a policy against AI usage I won't submit AI-generated code because I respect your decision.
That's just not the point.
The industrywide tsunami of tech debt arising from AI detritus[1] will be interesting to watch. Tech leadership is currently drunk on improved productivity metrics (via lines of code or number of PRs), but I bet velocity will slow down, and products be more brittle due to extraneous AI-generated, with a lag, so it won't be immediately apparent. Only teams with rigorous reviews will fare well in the long term, but may be punished in the short term for "not being as productive" as others.
1. From personal observation: when I'm in a hurry, I accept code that does more than is necessary to meet the requirements, or is merely not succinct. Where as pre-AI, less code would be merged with a "TBD" tacked on
I also have no idea what the social norms are for AI. I posted the comment after a friend on Discord said I should disclose my use of AI.
The underlying purpose of the PR is ironically because Cline and Copilot keep trying to use `int` when modern C++ coding standards suggest `size_t` (or something similar).
On top of all that every open source project has a gray hair problem.
Telling people excited about a new tech to never contribute makes sure that all projects turn into templeOS when the lead maintainer moves on.
I accept that some projects allow this, and if they invite it, I guess I can’t say anything other than “good luck,” but to me it feels like long odds that any one contributor who starts out eager to make others wade through enough code to generate that many comments purely as a one-sided learning exercise will continue to remain invested in this project to the point where I feel glad to have invested in this particular pedagogy.
No you don't. And if you're that entitled to people's time you will simply get no new contributors.
Trying to understand a problem and taking some time to work out a solution proves that you’re actually trying to learn and be helpful, even if you’re green. Using a LLM to generate a nearly-thousand-line PR and yeeting it at the maintainers with a note that says “I don’t really know what this does” feels less hopeful.
I feel like a better use of an LLM would be to use it for guidance on where to look when trying to see how pieces fit together, or maybe get some understanding of what something is doing, and then by one’s own efforts actually construct the solution. Then, even if one only has a partial implementation, it would feel much more reasonable to open a WIP PR and say “is this on the right track?”
And you can't go and turn this around into "but the gate keeping!" You just said that expecting someone to learn and be an asset to a project is entitlement, so by definition someone with this attitude won't stick around.
Lastly, the reason that the resume builder wants the "LLVM contributor" bullet point in the first place is precisely because that normally takes effort. If it becomes known in the industry that getting it simply requires throwing some AI PR over the wall - the value of this signal will quickly diminish.
While I legitimately do find templeOS to be a fascinating project, I don’t think there was anything to learn from it at a computer science level other than “oh look, an opinionated 64-bit operating environment that feels like classical computing and had a couple novel ideas”
I respect that instances like it are demonstrably few and far between, but don’t entertain its legacy far beyond that.
I disagree, actually.
I think that his approach has a lot to teach aspiring architects of impossibly large and complex systems, such as "create a suitable language for your use-case if one does not exist. It need not be a whole new language, just a variation of an existing one that smooths out all the rough edges specific to your complex software".
His approach demonstrated very large gains in an unusually complicated product. I can point to projects written in modern languages that come nowhere close to being as high-velocity as his, because his approach was fine-tuned to the use-case of "high-velocity while including only the bare necessities of safety."
It seems a little mean to tell him to stop coding forever when his intentions and efforts seem pretty positive for the health of the project.
This means that he did not put serious effort into understanding what, when and why others do in a highly structured project like LLVM. He "wrote" the code and then dumped "written" code into community to catch mistakes.
There are pitfalls everywhere. It’s not so small that you can get everything in your head with only a reading. You need to actually engage with the code via contributions to understand it. 100+ comments is not an exceptional amount for early contributions.
Anyway, LLVM is so complex I doubt you can actually vibcode anything valuable so there are probably a lot of actual work in the contribution.
There is a reason the community didn’t send them packing. Onboarding new comer is hard but it pays off.
> Have you ever contributed to a very large project like LLVM?
> I would say clearly not from the comment.
> It’s not so small that you can get everything in your head with only a reading.
So, after reading code, one should write down what made him amazed and find out why it is so - whether it is a custom of a project or a peculiarity of code just read.
I actually have such a list for my work. Do you?
> You need to actually engage with the code via contributions to understand it. 100+ comments is not an exceptional amount for early contributions.
That PR we discuss here contains a maintainer's requirement to remove excessive commenting - PR's author definitely did not do a codebase style matching cleanup job on his code before submission.
Sorry but that’s delusional.
The amount of people actually able to meaningfully read code, somehow identify what was so incredible it should be analysed despite being unfamiliar with the code base, maintain a list of their own likely error and self review is so vanishingly low it might as well not exist.
If that’s the bare a potential new contributor has to cross, you will get exactly none.
I’m personally glade LLVM disagree with you.
>The amount of people actually able to meaningfully read code, somehow identify what was so incredible it should be analysed despite being unfamiliar with the code base, maintain a list of their own likely error and self review is so vanishingly low it might as well not exist.
LLVM, I just checked, does not have a formal list of code conventions and/or typical errors and mistakes. Could they have that list, we would not have the pleasure to discuss that. That PR we are discussing would be much more polished and there would be much less than several dozens of comments.
> If that’s the bare a potential new contributor has to cross, you will get exactly none.
Be kind. Don't be snarky. Converse curiously; don't cross-examine. Edit out swipes.
When disagreeing, please reply to the argument instead of calling names. "That is idiotic; 1 + 1 is 2, not 3" can be shortened to "1 + 1 is 2, not 3."
Please don't fulminate. Please don't sneer, including at the rest of the community.
Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith.
Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something.
1. Good for you.
2. Ignore the haters in the comments.
> my latest PR is my second-ever to LLVM and is an entire linter check.
That is so awesome.
> The code is of terrible quality and I am at 100+ comments on my latest PR.
The LLVM reviewers are big kids. They know how to ignore a PR if they don't want to review it. Don't feel bad about wasting people's time. They'll let you know.
You might be surprised how many PRs even pre-LLMs had 100+ comments. There's a lot to learn. You clearly want to learn, so you'll get there and will soon be offering a net-positive contribution to this community (or the next one you join), if you aren't already.
Best of luck on your journey.
> You might be surprised how many PRs even pre-LLMs had 100+ comments
How well does that scale as the number of such contributions increases and the triage process itself becomes a sizable effort?
LLMs can inadvertently create a sort of DDoS even with the best intentions, and mitigating it costs something.
I sort of doubt that all of a sudden there's going to be tons of people wanting to make complex AI contributions to LLVM, but if there are just ban them at that point.
This is a different decision made by the LLVM project than the one made by Gentoo, which is neither right nor wrong IMHO.
> The code is of terrible quality and I am at 100+ comments on my latest PR.
This may be part of the justification of the published Gentoo policy. I am not a maintainer of same so cannot say for certain. I can say it is implied within their policy:
At this point, they pose both the risk of lowering the
quality of Gentoo projects, and of requiring an unfair
human effort from developers and users to review
contributions ...
Hence the Gentoo policy.
> ... but I would say it can be a fair tradeoff, because I'm learning quicker and can contribute at a level I otherwise couldn't.
I get it. I really do.
I would also ask - of the requested changes reviewers have made, what percentage are due to LLM generated changes? If more than zero, does this corroborate the Gentoo policy position of:
Popular LLMs are really great at generating plausibly
looking, but meaningless content.
I would also ask - of the requested changes reviewers have made, what percentage are due to LLM generated changes? If more than zero, does this corroborate the Gentoo policy position of "Popular LLMs are really great at generating plausibly looking, but meaningless content."
Previous codebases I've worked on during internships linted the first two in CI. And the documentation being formatted incorrectly is because I hand-wrote it without AI.
Out of the AI-related issues that I didn't catch, the biggest flaws were redundant comments and the use of string manipulation/parsing instead of AST manipulation. Useless comments are very common and I've gotten better at pruning them. The AI's insistence on hand-rolling stuff with strings was surprising and apparently LLVM-specific.
However, there was plenty of erroneous and invalid behaviour in the original AI-generated code, such as flagging `uint32_t` because the underlying type was an `unsigned int` (which wouldn't make sense as we want to replace `unsigned int` with `uint32_t`).
I prevented most of this from reaching the PR by writing good unit tests and having a clear vision of what the final result should look like. I believe this should be a basic requirement for trying to contribute AI-generated code to an open-source project but other people might not share the same belief.
> However, there was plenty of erroneous and invalid behaviour in the original AI-generated code ...
> I prevented most of this from reaching the PR by writing good unit tests and having a clear vision of what the final result should look like.
This identifies an interesting question in my mind:
If an LLM code generator is used, is it better to use
it for generating production code and writing tests
to verify or write production code and use LLM
generated code to produce tests to verify?
Maybe this question would be a good candidate for an "Ask HN".
> I believe this should be a basic requirement for trying to contribute AI-generated code to an open-source project but other people might not share the same belief.
FWIW, I completely concur.
> If an LLM code generator is used, is it better to use it for generating production code and writing tests to verify or write production code and use LLM generated code to produce tests to verify?
I do both.
1. Vibe code the initial design with input on the API/architecture.
2. Use the AI to write tests.
3. Carefully scrutinize the test cases, which are much easier to review than the code.
4. Save both.
5. Go do something else and let the AI modify the code until the tests/linting/etc passes.
6. Review the final product, make edits, and create the PR.
The output of step 1 is guaranteed to be terrible/buggy and difficult to review for correctness, which is why I review the test cases instead because they provide concrete examples.
Step 5 eliminates most of the problems and frees me to review important stuff.
The whole reason I wrote the check is because AI keeps using `int` and I don't want it to.
I think there are a couple of good signals in what you've said but also some stuff (at least by implication/phrashing) that I would be mindful of.
The reason why I think your head is fundamentally in a good place is that you seem to be shooting for an outcome where already high effort stays high, and with the assistance of the tools your ambition can increase. That's very much my aspiration with it, and I think that's been the play for motivated hackers forever: become as capable as possible as quickly as possible by using every effort and resource. Certainly in my lifetime I've seen things like widely distributed source code in the 90s, Google a little later, StackOverflow indexed by Google, the mega-grep when I did the FAANG thing, and now the language models. They're all related (and I think less impressive/concerning to people who remember pre-SEO Google, that was up there with any LLM on "magic box with reasonable code").
But we all have to self-police on this because with any source of code we don't understand, the abstraction almost always leaks, and it's a slippery slope: you get a little tired or busy or lazy, it slips a bit, next thing you know the diff or project or system is jeopardized, and you're throwing long shots that compound.
I'm sure the reviewers can make their own call about whether you're in an ok place in terms of whether you're making a sincere effort or if you've slipped into the low-integrity zone (LLVM people are serious people), just be mindful that if you want the most out of it and to be welcome on projects and teams generally, you have to keep the gap between ability and scope in a band: pushing hard enough to need the tools and reviewers generous with their time is good, it's how you improve, but go too far and everyone loses because you stop learning and they could have prompted the bot themselves.
> and especially for not mentioning that it's AI generated code
https://github.com/llvm/llvm-project/pull/146970#issuecommen...
irony really is dead
I find that latter part particularly relevant, considering the hoopla is about AI bros being lazy dogs who can't be bothered to put in the hard work before attempting to contribute. Irony being then that the person above just took an intentionally cut short citation to paint the person in a somehow even more negative light than they'd have otherwise appeared in, while simultaneously not even bothering to review the conduct they're proposing to police to confirm it actually matches their knowingly uncharitable conjecture. Two wrongs not making a right or whatever.
It should be clear that my objection is to the mix of coc + ai in the context of llvm, not to this specific instance where someone is acting within the rules llvm has written down.
It's literally just an opinion.
> It's literally just an opinion.
The definition of "refute"[0] is:
to prove wrong by argument or evidence : show to be false or erroneous
You may notice that opinions are like assholes: everyone has theirs. They're literally just "thoughts and feelings". They may masquerade as arguments from time to time, much to my dismay, but rest assured: there's nothing to "refute", debate, or even dispute on them. Not in general, nor in this specific case either.
Consider:
"This show is shit. The pacing is terrible, and the writing is cringe. Nobody should watch this garbage."
A reasoning most difficult to refute indeed. It's shit, because the person thinks it's shit, just in two other, more specific ways. And nobody should watch it, because they think it's shit, and (presumably) think that shit shows shouldn't be watched. A most terrific derivation of all times for sure.
Thinking an opinion can be refuted is analogous to thinking that definitions can be proven false. Which is to say, utterly misguided and plain wrong. They (opinions) are assertions of belief, with any use of logic in them just being rhetorical garnish, if not outright a diversion.
As a simple example, suppose X is a set and r is a relation on X. If I define Y := X/r , the set of equivalence relations with respect to r, this implicitly assumes that the relation r is an equivalence relation.
Maybe. There's a known condition in pigs that prevents them from forming one.
1 - Publicity stunt
In an effort to get more attention for the Gentoo project,
the maintainers created an outlandish policy to drive
traffic. This would seem unlikely due to the policy
decision being voted upon over a year ago.
2 - Fear of LLM's replacing Gentoo maintainers
This appears to not be the case based on the Gentoo minutes[0] provided:
Policy on AI contributions and tooling
======================================
Motion from the email thread:
> It is expressly forbidden to contribute to Gentoo any content that has
> been created with the assistance of Natural Language Processing
> artificial intelligence tools. This motion can be revisited, should
> a case been made over such a tool that does not pose copyright, ethical
> and quality concerns.
The vote was 6y/0n/1a (all present members voted yes).
sam noted as obiter dicta that the mail also mentioned:
> This explicitly covers all GPTs, including ChatGPT and Copilot, which is
> the category causing the most concern at the moment. At the same time,
> it doesn't block more specific uses of machine learning to problem
> solving.
Several council members noted that we will revisit the policy if and
when circumstances change and that it isn't intended to permanent,
at least not in its current form.
3 - Experience with LLM-based change requests
If the policy is neither a publicity stunt nor fear of
LLM's replacing maintainers, then the simplest explanation
remaining which substantiates the policy is maintainers
having experience with LLM use and then publishing their
decisions therein.
You wrote:
You may notice that opinions are like assholes: everyone
has theirs. They're literally just "thoughts and feelings".
They may masquerade as arguments from time to time, much to
my dismay, but rest assured: there's nothing to "refute",
debate, or even dispute on them. Not in general, nor in
this specific case either.
I am not a Gentoo maintainer so cannot definitively state possibility #3 is what happened. Maybe one or both of the other two possibilities is what transpired. I doubt it, but if you have evidence refuting possibility #3, please share so we may all learn.
In order to "refute" their entire position, if we accept that to even make sense (I do not), I'd need to either prove them wrong about what their opinions are (nonsense), or show evidence they were actually holding a different opinion that ran contrary to what they shared (impossible, their actual opinion is known only to them, if that). There's very little "logical payload" to their published policy, if any. It's a series of opinions, and then a conclusion. Hence my example with the person not liking a given TV show, but stating their distaste as a fact of the world.
> I doubt it, but if you have evidence refuting possibility #3, please share so we may all learn.
Why am I being rhetorically coerced into engaging with something from a false set of options of your imagination, exactly?
That position would look better if they hadn't relied so heavily on feelings to justify the announcement:
>> Their operations are causing concerns about the huge use of energy and water.
>> The advertising and use of AI models has caused a significant harm to employees [which ones?] and reduction of service quality.
>> LLMs have been empowering all kinds of spam and scam efforts.
There is no experience or rational discourse involved there.
4 - There is a very active anti-LLM activist movement and they care more about participating in it than they care about free software.
For example, see their rationale, which are just canned anti-LLM activist talking points. You see the same ones repeated and memed ad nauseam if you lurk on anti-AI spaces.
> 4 - There is a very active anti-LLM activist movement ...
All I can say to this is that my position is Large Language Models (LLM's) are a combination of algorithms and data.
As as such, for me they do not qualify as anything to be either "pro" or "anti", let alone a participant of an activist movement.
>> As as such, for me they do not qualify as anything to be either "pro" or "anti", let alone a participant of an activist movement.
> They were not talking about LLMs being participants of anything ...
Clearly I was referencing LLM's being something to foment "an activist movement" in an attempt to de-escalate the implication of there being some kind of "anti-LLM activist movement."
> ... but people who are against LLMs in whatever capacity. Surely people can be participants of a movement.
At this point your replies to my posts appear to be intentionally adversarial.
Well, no, that really wasn't clear to me at all. I don't think it was clear in general either.
> At this point your replies to my posts appear to be intentionally adversarial.
Not my actual intention, apologies, although I 100% understand if at this point that is not at all believable.
Are you trying to imply that there isn't an anti-LLM activist movement? There certainly is
you can just google it
https://www.google.com/search?hl=en&q=anti%20llm%20activist%...
a subreddit dedicated to the anti-AI movement
https://www.reddit.com/r/antiai/
"what is the goal of the so-called anti-ai movement"
https://www.reddit.com/r/aiwars/comments/1mn3hh8/what_is_the...
"how do we build an anti-ai movement"
https://www.reddit.com/r/ArtistHate/comments/1hk3wuj/how_do_...
"The New Luddites Aren’t Backing Down
Activists are organizing to combat generative AI and other technologies—and reclaiming a misunderstood label in the process."
https://www.theatlantic.com/technology/archive/2024/02/new-l...
"Anti-AI Explained: Why Resistance to Artificial Intelligence Is Growing
As AI tools become more advanced, a growing chorus of critics is raising alarms about job loss, misinformation and other societal risks. Learn what’s fueling the anti-AI movement."
https://builtin.com/artificial-intelligence/anti-ai
"together against AI"
https://togetheragainstai.com/
and on and on
I invite people to become familiar with the movement and their arguments and compare them to the Gentoo rationale.
It’s just what every other tech bro on here wants to believe, that using LLM code is somehow less pure than using free-range-organic human written code.
No, made. Which is a very important difference.
Popular LLMs are really great at
generating plausibly looking, but meaningless
content. They are capable of providing good
assistance if you are careful enough
Humans are capable of generating extremely poor code. Improperly supervised LLMs are capable of generating extremely poor code.
How is this is an LLM-specific problem?
I believe part of (or perhaps the entire) the argument here is that LLMs certainly enable more unqualified contributors to generate larger quantities of low-quality code than they would have been able to otherwise. Which... is true.
But still I'm not sure that LLMs are the problem here? Nobody should be submitting unexpected, large, hard-to-review quantities of code in the first place, LLM-aided or otherwise. It seems to me that LLMs are, at worst, exposing an existing flaw in the governance process of certain projects?
As for humans who can’t write code, their code doesn’t tend to look like they can.
Without LLMs, people are less likely to submit such PRs. With LLMs they're more likely to do so. This is based on recent increases in such PRs pretty much all projects have seen. Current LLMs are extremely sycophantic & encourage people to think they're brilliant revolutionary thinkers coming up with the best <ideas, code, etc> ever. Combined with the marketing of LLMs as experts it's pretty easy to see why some people fall for the hype & believe they're doing valuable work when they're really just dumping slop on the reviewers.