I think the real power will come when generative models get combined with e.g. refinement types, which are more or less analogous to contracts. Imagine, you decompose a problem into some functions with some contracts, and you get implementations for free. Plus, they will be guaranteed to match the specification!

I was discussing this with GitHub when they were hiring for Copilot, but understandably they wanted to get the basic functionality right first. I think it is the next step, and a very interesting topic for a startup or OpenAI et al. to tackle. It has the potential to make programming both more robust and faster, possibly bringing us closer to the correctness levels of classical engineering disciplines.

Similarly, Stephen Wolfram elaborated in a Lex Fridman interview on a combination of having GPT express computational language (like Wolfram Alpha).

Me too. I'm writing a compiler for fun and its extremely helpful to have it auto complete entire simple functions, like converting AST nodes to a string representation for example.

Doesn't it defeat the purpose of writing a compiler for fun if ChatGPT is going to write most of it?

I don't think the fun part is actually pressing on the keyboard.

Never done it myself, but I imagine understanding how the components all fit together is more valuable/fun than dealing with the minutiae of every function.

a more intuitive search as well. i am going to fly to paris in april and wanted to know what time it would be in denver when i landed. chatgpt was better-suited to the task than a multi-step duckbingle search

A Google search could do that 20 years ago.

I personally will never use the flight time zone related queries anymore. Almost like it didn’t know how time zones worked.

Yeah I think this captures my experience well. Not so much copilot as administrative assistant for my editor.

i find the inline suggestions of copilot distracting in any but the most mundane of cases. I know you can disable them but it still feels like it should be enabled instead of hotkeyed

This feels more or less in line with what my team has found, we gave everyone a copilot seat in our Github org, and anecdotally speaking it feels like we've seen roughly a 5-10% increase in productivity. This is of course self reported and not measured against any metrics. Assuming we're right about that, its an easy sell when we account for what our internal hourly rate is.

we also found the same as the OP, It's good for simple problems or boilerplate, not great for more complex problems.

what is the cost of analyzing the proposed solution and fixing it so it does what you want and how does that compare to not using AI and writing the code yourself?

That's an excellent question. I'm typically able to stay focused on a task in spite of distractions, but working with generated nonsense while coding something new has derailed me quite a few times.

The novelty has definitely worn off for me, at least.

I use copilot and write a whole lot of code every day. It's very very hard for me to believe anyone could get a 5-10% boost in productivity. It's often incorrect and when it is correct, it's often trivial.

It is a value add, but I'd put it closer to 0.5% if that. Over, say, 8 hours of coding time, it might save me a couple of minutes total.

Which, from a company expense perspective is still worth it, but an order of magnitude less than your anecdote.

It's very hard for me to envision how copilot could save someone that kind of time.

Interesting, the experience is very different for me. What language are you working with? I find it to be pretty strong in my Python and JS projects.

Many, including python + js. What does it provide? How do you use it?

I'm not diagnosed but I almost certainly have something like ADHD. It feels like my brain will take the most insignificant excuse to start paying attention to something else. Copilot boosts my productivity by 25-30%, easily, by removing many of these "excuses".

I am an experienced programmer, but sometimes I'll have a clear intention in my head of what I want to do, and the knowledge/experience of what code would accomplish that thing, and I still back away from it and start looking at something else, whether work, social media, whatever. Writing this, I know it might sound ridiculous or lazy, but it's true.

Copilot bridges this gap, often miraculously. The gap between thought and code is shortened, and the windows of time where I might lose focus seem to be drastically shortened.

For me, the key is that I do know how to write most of the code that Copilot writes for me, I'm just not good at actually writing it, or at least doing so in a sustained, consistent way.

Super interesting perspective and insight. I really appreciate you chiming in. Hadn't considered something like this.

I can relate to that experience.

I feel like a lot of this is just that ADHD brains (and brains in general) want to work at the right level of abstraction. If I have to add headers to 20 word docs, it’s probably a bit quicker to do that manually than to write a script to do it, but I write the script anyway because the alternative is too boring. A lot of code is like that too, and “getting the AI to understand what I’m trying to get it to do” is more interesting.

Is it worth paying 20$ for 5% improvement

Unless your time is borderline worthless, obviously yes.