When I am able ask a very simple question of an LLM which then prevents me having to context-switch to answer the same simple question myself; this is a big time saver for me but hard-to-quantify.
Anything that reduces my cognitive load when the pressure is on is a blessing on some level.
My issue with most AI discussions is they seem to completely change the dimensions we use to evaluate basic things. I believe if we replaced "AI" with "new useful tool" then people would be much more eager to adopt it.
What clicked for me is when I started treating it more like a tool and less like some sort of nebulous pandora's box.
Now to me it's no different than auto completing code, fuzzy finding files, regular expressions, garbage collection, unit testing, UI frameworks, design patterns, etc. It's just a tool. It has weaknesses and it has strengths. Use it for the strengths and account for the weaknesses.
Like any tool it can be destructive in the hands of an inexperienced person or a person who's asking it to do too much. But in the hands of someone who knows what they're doing and knows what they want out of it - it's so freakin' awesome.
Sorry for the digression. All that to say that if someone believes it's a productivity boost for them then I don't think they're being misled.
where programmers are learning this idea:
that writing code fast is ideal.
If it takes 30 years to write one loc, it takes 30 years.
Ideally, it takes 30 years to write zero lines of code.
more logic does not validate the truth.
>that writing code fast is ideal.
At places that pay them.
High level design, rough outlines and approaches, is the worst place to use AI. The other place AI is pretty good is surfacing api call or function calls you might not know about if you're new to the language. Basically, it can save you a lot of time by avoiding the need for tons of internet searching in some cases.
Unit tests actually need to be correct, down to individual characters. Same goes with API calls. The API needs to actually exist.
Contrast that with "high level design, rough outlines". Those can be quite vague and hand-wavy. That's where these fuzzy LLMs shine.
That said, these LLM-based systems are great at writing "change detection" unit tests that offer ~zero value (or negative).
These tests are also very time consuming to write, with lots of boilerplate that AI is very good at writing.
"speaks volumes" lol
Hard to describe, easy to spot.
Some people write tests that are tightly coupled to their particular implementation.
They might have tons of setup code in each test. So refactoring means each test needs extensive rewrites.
Or there will be loads of asserts that have little to do with the actual thing being tested.
These tests usually have negative value as your only real option as another developer is to simply delete them all and start again.
That's what I would interpret the GP as meaning when they use the phrase "change detection" tests.
That is not due to people choice but due to what actual code being tested does.
I think integration tests and end to end tests are much better.
But code should change. What shouldn't change, if business rules don't change, is APIs and contracts. And for that we have integration tests and end to end tests.
I will start to care when integration tests are failing, because that is an actual bug. Then I will fix the bug and move over.
That’s not at all true in my experience. With minimal guidance they put out pretty sensible tests.
Yes and no. They get out all the initial annoying boilerplate of writing tests out of the way, and the tests end up being mostly decent on the surface, but I have to manually tweak the behavior and write most of the important parts myself, especially for non-trivial tricky scenarios.
However, I am not saying this as a point against LLMs. The fact that they are able to get a good chunk of the boring boilerplate parts of writing unit tests out of the way and let me focus on the actual logic of individual tests has been noticeably helpful to me, personally.
I only use LLMs for the very first initial phase of writing unit tests, with most of the work still being done by me. But that initial phase is the most annoying and boring part of the process for me. So even if I still spend 90% of the time writing code manually, I still am very glad for being able to get that initial boring part out of the way quickly, without wasting my mental effort cycles on it.
I'd rather see the need for integration tests and end to end tests. I want to test business logic not assert that 2 + 2 = 4.
Once you have a way to do a good integration/e2e test, the results of the constituent unit tests don't provide as much value.
I'd rather run one big complicated thing for real once than hang my hat on a bunch of fake green checkmarks that update every 50 milliseconds.
> I un-did your change which made no sense to me and now everything is broken, why is what you did necessary?
I think we just have to ask ourselves what we want it to be good at, and then be diligent about generating decades worth of high quality training material in that domain. At some point, it'll start getting the details right.
Anyhow, it seems to me like it is working. It's just working better for the really old stuff because:
- there has been more time for training data to accumulate
- some of it predates the trend of monetizing data, so there was less hoarding and more sharing
It may be that the hard slow way is the only way to get good results. If the modern trends re: products don't have the longevity/community to benefit from it, maybe we should fix that.
Ask it to review its own code for any problems?
Also identify typical and corner cases and generate tests?
Question marks here because I have not used the tool.
The size & depth of each accepted code step is still up to the developer slash prompter
There have been a bunch of times where I've asked it to write me a snippet of code, and it cheerfully gave me back something that doesn't work for one reason or another. Hallucinated methods are common. Then I ask it to check its code, and it'll find the error and give me back code with a different error. I'll repeat the process a few times before it eventually gets back to code that resembles its first attempt. Then I'll give up and write it myself.
As an example of a task that it failed to do: I asked it to write me an example Python function that runs a subprocess, prints its stdout transparently (so that I can use it for running interactive applications), but also records the process's stdout so that I can use it later. I wanted something that used non-blocking I/O methods, so that I didn't have to explicitly poll every N milliseconds or something.
I can see where you are coming from.
Maintaining a better creative + technical balance, instead of see-sawing. More continuous conscious planning, less drilling.
Plus the unwavering tireless help of these AI's seems psychologically conducive to maintaining one's own motivation. Even if I end up designing an elaborate garden estate or a simpler better six-axis camera stabilizer/tracker, or refactoring how I think of primes before attempting a theorem, ... when that was not my agenda for the day. Or any day.
The apparent speed up is mostly a deception. It definitely helps with rough outlines and approaches. But, the faster you go, the less you will notice the fine details, and the more assumptions you will accumulate before realizing the fundamental error.
I'd rather find out I was wrong within the same day. I'd probably have written some unit tests and played around with that function a lot more if I had handcrafted it.