An eye opening example for me was that I was working with a Ruby/Rails class that was testing various IP configurations [1] and I was able to just copy and paste it into chatgpt and say "write some tests for this".

It wasn't really anything I couldn't have written in a half hour or so but it was so much faster. The real kicker is that by default chatgpt wrote Rspec and I was able to say "rewrite that in minitest" and it worked.

1 - https://wafris.org/ip-lookup

I can't speak for OP, but for me I literally never use stack overflow any more, and I spend about 90% less time on Google

Yep exactly. I guess I haven't hit the 25 messages in 3 hours limit, but whenever there's an API or library I'm not familiar with. I can get my exact example in about 10 seconds from ChatGPT 4

Are those popular APIs?

I've found Copilot useful when writing greenfield code, but very unhelpful generating code that uses APIs not popular enough to have significant coverage on StackOverflow. Even if I have examples of correct usage in the same file it still guesses plausible but wrong types.

I haven't bought GPT 4 but I'm curious if it's much better at this.

If you don't mention a library by name it is liable to make something up by picking a popular library in another language and converting the syntax to the language you asked for.

If you ask for something impossible in a library it will also frequently make up functions or application settings. If you ask for something obscure but hard to do, it might reply that it's impossible but it is possible if you know how and teach it.

I sort of compare prompt engineering to Googling - you sometimes have to search for exactly the right terms that you want to appear in the result in order to get the answer you're looking for. It's just that the flexibility of ChatGPT in writing a direct response sometimes means it will completely make up an answer.

There's also a limitation that the web interface doesn't actually let you upload files and has a length limit for inputs. For Copilot, I'm looking forward to Copilot X: https://www.youtube.com/watch?v=3surPGP7_4o

This was neither. I've forgotten the exact words I typed but it was something like this.

Prompt:

    fn encode(value: Foo) {
        capnproto::serialize_packed:: serialize_message(value);
    }

    fn decode(input: &[u8]) {

Expected:

   capnproto::serialize_packed:: deserialize_message(input);

Generated

     capnproto::PackedMessageDeserializer::deserialize(input)

Maybe a difference between prompts or v3.5 vs v4. I asked ChatGPT v4 and it gave an answer that used capnproto::serialize_packed::read_message. I asked it about the difference between read_message and deserialize_message and this is the response it gave (truncated, as it got some parts wrong):

> In summary, you should use read_message when working with packed messages and deserialize_message when working with unpacked messages. Make sure you choose the appropriate serialization and deserialization functions based on the format in which your messages are stored or transmitted.

Google was surprisingly little help on the topic. At best it pointed me to https://capnproto.org/encoding.html#packing which covers the general idea but glosses over parts.

The problem with ChatGPT is you can never be certain or confident that its answers are correct. It’s as useful as rolling dice in guessing a number sometimes. With the training from the internet, the dice are loaded, but the answer is still likely to be wrong because it just assembles words together.

Are you not fact checking chatgpt? I've seen wrong info, especially subtle things. It seemed reckless to use as-is.

For fun I tried asking chatgpt to create a simple using an opensource project I maintain. The generated answer was sort of correct but not correct enough to copy and paste. It missed including a plugin, used a version of the project that doesn't exist yet, and generated data that wasn't valid datetimes.

Both ChatGPT and StackOverflow suffer from content becoming outdated. So some highly-upvoted answer on StackOverflow has been out of date since 2011, and now ChatGPT is trained on it.

I see the future as writing test cases (perhaps also with ChatGPT), and separately using ChatGPT to write the implementation. Perhaps we will just give it a bunch of test cases and it will return code (or submit a PR) that passes those tests.

Curious if that’s because AI provides better answers?

It’s certainly not quicker answers, right?