News for hackers. Stuff that matters.
Re:40% of the code GitHub Copilot users check-in is AI gener... by luxurytent on Tuesday March 14, 2023 @09:13PM (#35162319)
I'm not surprised. A lot of the code I'll write in a day is trivial, calling helpers, writing test scaffolding, etc. I'm still building cool things the other 60% of time, but this job as we know it comes with a lot of repetition. (esp. if you write unit tests ;-))
Re:40% of the code GitHub Copilot users check-in is AI gener... by aliencat on Tuesday March 14, 2023 @09:27PM (#35162436)
unit test definitely a great use of copilot.
Re:40% of the code GitHub Copilot users check-in is AI gener... by wokwokwok on Tuesday March 14, 2023 @09:38PM (#35162542)
What are you talking about?

I don't know anyone who uses it for this; specifically tests are really bad if they're subtly wrong.

Maybe to scaffold the test function, but the actual test if completely useless if you don't trust it.

So like... generate code and have robust tests, or write robust code... but, it's really really daft to generate tests that might hallucinate some random crap (and copilot really does sometimes).

Other people have said this, but copilot is auto complete.

You use it every time you press tab.

Do you accept an auto-complete suggestion 40% of the time? ...mmmm, yes, well, guess why 40% of code is generated by copilot.

It's not because people are generating tests with it.

Re:40% of the code GitHub Copilot users check-in is AI gener... by rjh29 on Tuesday March 14, 2023 @09:51PM (#35162653)
Most of the time I know exactly what tests I want to write but it's annoying to write them. Copilot autocompletes, I check it and then I use it.
Re:40% of the code GitHub Copilot users check-in is AI gener... by Waterluvian on Tuesday March 14, 2023 @09:54PM (#35162692)
For me the worst part about this is that writing tests tediously makes me reach for layers of abstraction on my test code. And then suddenly my test code is complicated and needs its own test code, and changing a test can often be problematic to the abstractions I foolishly employed.

Being able to churn out the boilerplate for tests, which can make DRY a non-concern, is great. I just hope that if I do this, I never get sloppy, and I always review the tests.

Re:40% of the code GitHub Copilot users check-in is AI gener... by rjh29 on Wednesday March 22, 2023 @10:06AM (#35261174)
I agree! I try not to make test code too complicated, and prefer repetitive code over abstractions (else you have to test the tests!) so Copilot is useful there.
Re:40% of the code GitHub Copilot users check-in is AI gener... by dandellion on Tuesday March 14, 2023 @09:52PM (#35162674)
In my case about 50% of the test code is just boilerplate, so I usually type the test name and generate the rest. Most of the time I have to rewrite the actual test logic (although very trivial tests are sometimes correct), but I probably keep more than 40% if we're talking tests specifically.
Re:40% of the code GitHub Copilot users check-in is AI gener... by paro_nej on Tuesday March 14, 2023 @10:09PM (#35162800)
> In my case about 50% of the test code is just boilerplate

Why not replace it with a function call?

Re:40% of the code GitHub Copilot users check-in is AI gener... by travisjungroth on Tuesday March 14, 2023 @10:15PM (#35162845)
A common occurrence is something like a pool of 10 actions where a bunch of tests each do 3 to 7 of them. This is very hard to abstract with a function call.
Re:40% of the code GitHub Copilot users check-in is AI gener... by aaomidi on Tuesday March 14, 2023 @10:12PM (#35162828)
Some languages are just noisy.

Go, Java I’ve experienced are noisier than others.

Re:40% of the code GitHub Copilot users check-in is AI gener... by lupire on Tuesday March 14, 2023 @11:39PM (#35163524)
That makes the test harder to read. Tests should be dumb.
Re:40% of the code GitHub Copilot users check-in is AI gener... by dandellion on Wednesday March 15, 2023 @09:33PM (#35177060)
In the case I was referring to, mainly to keep the code consistent with what was already there and the PRs small, as the code base is primarily owned by a different team. It's also pretty innocuous short tests that read well as they are.
Re:40% of the code GitHub Copilot users check-in is AI gener... by teaearlgraycold on Tuesday March 14, 2023 @09:57PM (#35162713)
I use it for unit tests. That's one of its best use cases. Obviously I read the code it writes down.
Re:40% of the code GitHub Copilot users check-in is AI gener... by kakwa_ on Wednesday March 15, 2023 @04:29AM (#35165396)
In a recent personal project (a custom archive format extractor), I have used chatgpt to generate tons of unit tests.

And honestly, I was extremely impressed. With a bit of context, it was able to generate almost correct test data for a fairly complex binary data format.

I've also leverage it heavily to

* generate doxygen comments

* get usage examples of libs I never used

* create a whole bunch of util functions.

And honestly for all these tedious tasks, it has done a far better job I would have done.

Comments were consistent in styling, the base examples where fairly good, and the util functions are really good, specially the error handling (which I would probably have semi-consciously skipped tbh).

In fairness, I modified most of this code slightly to make it fit my project structure, or tweak a bit what the IA didn't got right (it never fully understood some offset fields in the file format, but got pretty close to at times. And in fairness my naming for these offset fields was a bit questionable).

Heck, as a test, I even threw at it an RFC-like spec of the file format, and asked him to generate a python parser. The result was not 100 correct, but definitely a good start to iterate on.

In the end, this side project took me 2 weeks with chatgpt probably saving around 1 week of dev. And it also greatly helped improving the quality of the project (better doc, better tests).

Re:40% of the code GitHub Copilot users check-in is AI gener... by bananapub on Wednesday March 15, 2023 @04:42AM (#35165509)
> I don't know anyone who uses it for this

that would be surprising - it does a fantastic job of producing e.g. dumb unit tests. for instance, Copilot + a Go table testing template means you can churn out the simple "make a request to this url with this data, ensure I get a 200 and the response contains 'id' and not 'error'" extremely quickly. the code is trivial but tedious, so you can quickly inspect for sanity and run to ensure they pass, then commit and have them checking future changes.

> it's really really daft to generate tests that might hallucinate some random crap (and copilot really does sometimes).

variations on this comment are all over these threads, which is bizarre. hallucinating means you waste a few seconds reading the code it produced, not that you commit incorrect code.

this isn't like ChatGPT advising people to drink bleach, Copilot is a dumb tool offering an expert (you) suggested solutions for your expert consideration

Re:40% of the code GitHub Copilot users check-in is AI gener... by wongarsu on Tuesday March 14, 2023 @10:03PM (#35162753)
I guess this could be true in some languages and settings, for example when a notable portion of your tests verify correct behavior with null values, arguments of wrong type, etc. When talking about unit tests that verify the actual logic, you should naturally become more careful with your copilot suggestions. Especially if copilot also wrote the code it's testing.
Re:40% of the code GitHub Copilot users check-in is AI gener... by cortesoft on Tuesday March 14, 2023 @10:12PM (#35162830)
Yeah, to me this doesn’t feel that different than using a library from another developer, just slightly more customized. Most code that is run is not written by the final developer, but by people further up the chain.
Re:40% of the code GitHub Copilot users check-in is AI gener... by js8 on Tuesday March 14, 2023 @11:27PM (#35163449)
The real question is, why we need to write "trivial" code? I think people have lost the plot there. We should be writing better compilers and libraries, instead of using AI that will write more code to deal with.

The obsession with code generation instead of creating proper abstractions is a scourge of today's programming. It's a regression in software engineering.

Re:40% of the code GitHub Copilot users check-in is AI gener... by layer8 on Tuesday March 14, 2023 @11:37PM (#35163509)
I agree to some extent. But there are redundancies that are inherently beneficial and make code more robust and readable, like separating interface from implementation (where some declarative elements have to be repeated), or explicit type annotations instead of full-program type inference. There are also strictly necessary redundancies, like when you pattern-match an ADT, you repeat its declared structure. For desirable or necessary redundancies like that, AI code completion can still save time.

Intelligent unit test creation (if it is indeed intelligent) is also a good use-case. Not to render thinking by the programmer unnecessary, but to save typing and to point out potentially useful tests that you might not have thought of.

Re:40% of the code GitHub Copilot users check-in is AI gener... by nine_k on Tuesday March 14, 2023 @11:44PM (#35163568)
This is trivial stuff which is handled well by any competent IDE from 2015, and even a well enough-working set of editor templates.

I don't see where "AI" is required for stuff like this.

Re:40% of the code GitHub Copilot users check-in is AI gener... by js8 on Tuesday March 14, 2023 @11:50PM (#35163612)
I agree, but also, this is actually where AI (whatever it is) would be helpful. Suggesting actual abstractions that could replace boilerplate. Of course, these can build on existing abstractions.
Re:40% of the code GitHub Copilot users check-in is AI gener... by lupire on Tuesday March 14, 2023 @11:37PM (#35163510)
The boilerplate makes it readable by humans.
Re:40% of the code GitHub Copilot users check-in is AI gener... by never_inline on Wednesday March 15, 2023 @01:38AM (#35164336)
I have made the same point elsewhere to AI enthusiasts friends of mine.

The fact that defining the same task in natural language to an AI is easier than writing it in formal language, means the formal language lacks expressiveness [0], vocabulary, or more likely, both.

But if they are both at same level of expressiveness and vocabulary, formal language wins, because it can spot and prevent ambiguity.

The question is how scalable it is to implement such vocabulary into formal language.

[0] - The line is often blury. How many languages allow to straightforwardly express `clamp m between 1.0 and 2.0`? You can argue this is vocabulary, or this is expressiveness.

Re:40% of the code GitHub Copilot users check-in is AI gener... by js8 on Wednesday March 15, 2023 @04:49AM (#35165554)
> How many languages allow to straightforwardly express `clamp m between 1.0 and 2.0`? You can argue this is vocabulary, or this is expressiveness.

And I think it would be a great place for AI in programming - automated refactoring. For example, it would see a code that clamps the value and could be replaced with a library function call clamp(m,1,2). (And it would also, presumably, made sure that the code is indeed the same.) So it could help you understand the vocabulary.

Or, if it sees you are doing this frequently, and it's not in the standard library, it would create the function.

But instead, this AI (more like artificial stupidity) will just suggest you the same thing you wrote because you don't know any better in your code base. That's not helping.

Re:40% of the code GitHub Copilot users check-in is AI gener... by never_inline on Wednesday March 15, 2023 @06:41AM (#35166197)
Good point. But I think current approach of large scale generative AI models is not sufficient for this.
Re:40% of the code GitHub Copilot users check-in is AI gener... by serpix on Wednesday March 15, 2023 @02:26AM (#35164625)
Sometimes it is just the syntax of your language. We can't all be writing lisp where the syntax is just another abstraction.
Re:40% of the code GitHub Copilot users check-in is AI gener... by js8 on Wednesday March 15, 2023 @04:39AM (#35165480)
> We can't all be writing lisp

Why not? Isn't it easier to learn new syntax than to figure out a correct prompting for AI? (Although actually, Haskell syntax would be something I would prefer to write in.)

Re:40% of the code GitHub Copilot users check-in is AI gener... by bananapub on Wednesday March 15, 2023 @04:38AM (#35165473)
> We should be writing better compilers and libraries

this is a bizarre claim to make and just completely ignores reality.

people ARE trying to make better compilers and libraries, and have been for decades. we are we are now after person-millennia of effort, and that still means the optima we're at requires writing some repetitive web template rendering code in JSX or whatever, which Copilot can automate. why didn't some homomorphic-monoid-transformer-auto-renderer win instead?

Re:40% of the code GitHub Copilot users check-in is AI gener... by js8 on Wednesday March 15, 2023 @04:59AM (#35165612)
I think the reality is that "if you build it they will come" is simply not always true, and worse is sometimes better. I don't think we are climbing to an optimum, at least unless it is culturally perceived that good abstractions are more valuable than just generating code.

The auto-renderers did actually win. There were GUI building tools at the end of 90s (like Visual Basic) that still put most web frameworks to shame. But, for various business reasons (mostly pricing, really) they were replaced with "web technology" - something open source but worse.

(And I am not dissing OSS, either. For example, Qt/KDE had some great abstractions that could serve as a basis for building applications. But the urge to reinvent the wheel has won.)

Re:40% of the code GitHub Copilot users check-in is AI gener... by circuit10 on Wednesday March 15, 2023 @05:10AM (#35165667)
I guess one difference is that Copilot can guess things that it’s not sure about and have you check over it, but a compiler can’t just guess what you want it it would be very unreliable
Re:40% of the code GitHub Copilot users check-in is AI gener... by js8 on Wednesday March 15, 2023 @08:10AM (#35167023)
Compilers do however guess that you don't want something - it's called compiler warnings. Linters do that as well. To me what you describe is more about how we communicate with compilers than an inherent limitation.
Re:40% of the code GitHub Copilot users check-in is AI gener... by tasuki on Wednesday March 15, 2023 @05:39AM (#35165792)
I couldn't agree more with everything you wrote. The problem with repetition is not that it's hard to generate, the problem is it's hard to maintain!

I've used ChatGPT to find libraries and to help me line up types. It was very helpful.