Yeah, I tried asking GPT-4 to write a simple Python program to find the millionth prime. It wrote the naive approach where you loop over numbers and for each number, check all of the possible divisors up to its square root, and then keep track of how many primes you've found. Obviously, this takes quite a while to execute for N = a million. And yet, if you ask GPT-4 what the output of the code will be, as if my magic (memorization) it immediately spits out 15485863. It makes it a lot more clear that it's not actually simulating the code.