News for hackers. Stuff that matters.
Re:OpenAI co-founder John Schulman says he will leave and jo... by lolinder on Tuesday August 06, 2024 @10:46AM (#41171334)
I've definitely tried that and it doesn't work for the problems I've tried. Claude's answers for me always have all the hallmarks of an LLM response: extremely confident, filled with misunderstandings of even widely used APIs, and often requiring active correction on so many details that I'm not convinced it wouldn't have been faster to just search for a solution by hand. It feels like pair programming with a junior engineer, but without the benefit of helping train someone.

I'm trying to figure out if I'm using it wrong or using it on the wrong types of problems. How do people with 10+ years of experience use it effectively?

Re:OpenAI co-founder John Schulman says he will leave and jo... by klyrs on Tuesday August 06, 2024 @12:12PM (#41172261)
I'm a mathematician and the problems I work on tend to be quite novel (leetcode feel but with real-world applications). I find LLMs to be utterly useless at such tasks; "pair programming a junior, but without the benefit" is an excellent summary of my experience as well.
Re:OpenAI co-founder John Schulman says he will leave and jo... by icholy on Tuesday August 06, 2024 @11:46AM (#41172006)
It's good for writing that prototype you're supposed to throw away. It's often easy to see the correct solution after seeing the wrong one.
Re:OpenAI co-founder John Schulman says he will leave and jo... by danielbln on Tuesday August 06, 2024 @10:54AM (#41171436)
I think the only way to answer that is if you can share an example of a conversation you had with it, where it broke down as you described.
Re:OpenAI co-founder John Schulman says he will leave and jo... by valval on Tuesday August 06, 2024 @12:37PM (#41172511)
For what I’m working on, I can also use the wrong approaches. Going through my fail often fail fast feedback loop is a lot more efficient with LLMs. Like A LOT more.

Then when I have a bunch of wrong answers, I can give those as context as well to the model and make it avoid those pitfalls. At that point my constraints for the problem are so rigorous that the LLMs lands at the correct solution and frankly writes out the code 100x faster than I would. And I’m an advanced vim user who types at 155 wpm.

Re:OpenAI co-founder John Schulman says he will leave and jo... by lolinder on Tuesday August 06, 2024 @01:43PM (#41173242)
> And I’m an advanced vim user who types at 155 wpm.

See, it's comments like this that make me suspect that I'm working on a completely different class of problem than the people who find value in interacting with LLMs.

I'm a very fast typer, but I've never bothered to find out how fast because the speed of my typing has never been the bottleneck for my work. The bottleneck is invariably thinking through the problem that I'm facing, trying to understand API docs, and figuring out how best to organize my work to communicate to future developers what's going on.

Copilot is great at saving me large amounts of keystrokes here and there, which is nice for avoiding RSI and occasionally (with very repetitive code like unit tests) actually a legit time saver. But try as I might I can't get useful output out of the chat models that actually speeds up my workflow.

Re:OpenAI co-founder John Schulman says he will leave and jo... by whackyMax on Tuesday August 06, 2024 @04:45PM (#41175154)
I have always thought of it as a way to figure out what doesn't work and get to a final design, not necessarily code. Personally, it's easy to verify a solution and figure out use cases that wouldn't work out. Keep on iterating until I have either figured out a mental model of the solution, or figured out the main problems in such a hypothetical solution.
Re:OpenAI co-founder John Schulman says he will leave and jo... by FLT8 on Tuesday August 06, 2024 @06:57PM (#41176358)
I'm sure I'm going to offend a bunch of people with this, but my experience has been similar to yours, and it reminds me of something "Uncle" Bob Martin once mentioned: the number of software developers is roughly doubling every two years, which means that at any given time half of the developer population has less than two years experience.

If you're an experienced dev, having a peer that enthusiastically suggests a bunch of plausible but subtly wrong things probably net-net slows you down and annoys you. If you're more junior, it's more like being shown a world of possibilities that opens your mind and seems much more useful.

Anyway, I think the reason we see so much enthusiasm for LLM coding assistants right now is the overall skew of developers to being more junior. I'm sure these tools will eventually get better, at least I hope they do because there's going to be a whole lot of enthusiastically written but questionable code out there soon that will need to be fixed and there probably won't be enough human capacity to fix it all.

Re:OpenAI co-founder John Schulman says he will leave and jo... by lolinder on Wednesday August 07, 2024 @12:19AM (#41178208)
Thanks for saying it explicitly. I definitely have the same sense, but was hoping someone with experience would chime in about use cases they have for it.