I had no idea GPT-3 could write code! It seems to do pretty well even with a one-sentence prompt:

  Write a C++ program that computes the dot product of two three-dimensional vectors.

---

  #include <iostream>
  #include <cmath>
  using namespace std;
  
  int main()
  {
      double x1, x2, x3, y1, y2, y3;
      
      //Get input for vector 1
      cout << "Enter x1: ";
      cin >> x1;
      cout << "Enter y1: ";
      cin >> y1;
      cout << "Enter z1: ";
      cin >> z1;
      
      //Get input for vector 2
      cout << "Enter x2: ";
      cin >> x2;
      cout << "Enter y2: ";
      cin >> y2;
      cout << "Enter z2: ";
      cin >> z2;
      
      //Calculate the dot product
      double dot_product = x1*x2 + y1*y2 + z1*z2;
      
      //Print the result
      cout << "The dot product is: " << dot_product << endl;
      
      return 0;
  }

EDIT: It even does Rust, I'm blown away! (I'll forgive the compile error, it's close enough)

  fn main() {
      let v1 = vec![1, 2, 3];
      let v2 = vec![4, 5, 6];

      let dot_product = v1.iter().zip(v2).map(|(x, y)| x * y).sum();

      println!("The dot product of {:?} and {:?} is {}", v1, v2, dot_product);
  }

In general, anything that has a “textbook” solution is easy. What it’s doing here is more recitation than synthesis. Where it becomes harder, and where my method is necessary, is when you need to specify the structure of the solution yourself because the model can’t get there on its own.

Exactly, that's why GPT3 generated things that were not included in task description in C++ example like reading the numbers first and then printing them. Same thing in Rust, here vectors are static, while in C++ you were asked. All because of examples on which the model was trained.

the reason it does well is because you gave it a question that might come up in coding interviews.

I think that's amazing on its own. Rather than having to do a leetcode hard on the fly, you can just ask for the solution. You'll still need to know the problem space in order to properly optimize it, but it can save a lot of time.

It’s incredible. I have used it to make several arduino programs for me, controlling different hardware and connecting them. It’s just amazing.

> Replace the $$dollar-quoted expressions$$ with the content described therein, or follow the directions they contain.

Is this a natural language meta-instruction prompt to GPT-3? If so, that seems kinda impressive. Does the model conceptualize this sentence somehow, or 'merely' recognize similar prompts from some specialized training?

Here's one of the craziest examples I've seen. It's just a page of instructions, and GPT-3 follows them https://mobile.twitter.com/goodside/status/15575245464120524...

I have just a question: how many prompts did it completely fail for each prompt it answered brilliantly? 5? 10? 100? That's what these "twitter threads showing gpt3 doing astonishing things" fail to show.

Many! In this example, my question was explicitly, “How many diverse tasks can I stack into a single generation before it becomes unreliable?” If you scroll down in the thread, I explain that these questions are on the “golden path” of tasks GPT-3 does well. There are any number of simple tasks I could have given it, like writing a sentence backwards or summing a list of 10 numbers, where it would fail every time.

Is there any way to get it to respond the same way when something is outside the golden path? So for example, if you gave it the backwards sentence task, it would respond with "I don't know how to do this" or really any way of programatically evaluating that it failed, without needing to know what the task itself was.

Knowing whether or not it’s giving you a sensible response is one of the things that are hard for gpt-3, unfortunately. It has no concept of failing.

On the contrary, doing a two-stage generation where the second stage simply judges whether a generation is correct can help a lot. It works even better if you give it several generations and let it choose whichever is the most truthful. I wrote a basic example of this here that uses my own confabulation-suppressing prompt in the first stage, but simpler variations of this exist in the published literature: https://twitter.com/goodside/status/1559586486705602562?s=21...

The hallucination-suppressing prompt it implicitly uses the output of is here: https://twitter.com/goodside/status/1556459121834168320?s=21...

Yes. You can, with effort, condition it to respond sensibly with phrases like “I’m sorry, I don’t know how to reverse strings,” or “I’m sorry, I can’t do any math calculation that a human couldn’t do in their head.” But in doing so you damage its ability to do some tasks it’s actually capable of, e.g. reciting a memorized answer to “What is the fourth root of 625?” Its memorization abilities are insane: It seems to know, for example, the exact MD5 hashes of all single-character alphanumeric strings. Much of the arithmetic it knows is probably similarly memorized, and it’s hard to clarify for it what aspects of that memory are safe to use.

The initial problem that got me interested in GPT-3 is suppressing confabulated answers to the Hofstadter-Bender questions published in The Economist. I eventually found an apparent solution but I’m yet to carefully validate it: https://twitter.com/goodside/status/1556459121834168320?s=21...

> Does the model conceptualize this sentence somehow, or 'merely' recognize similar prompts from some specialized training?

I don't think the model is big enough to Chinese Room prompts like this. It has to "conceptualize" them (in the least loaded sense of the word that works).

Nice! In general these are better if you run them at the lowest possible temperature. I.e., try temp=0 first for deterministic output and then raise slowly if you need to cherry-pick a better generation.