I gave ChatGPT a cryptographic signature scheme I developed and it found a way to forge signatures as my scheme was insecure. I thought ChatGPT was wrong because I didn't read the attack carefully enough and I didn't expect it be able to reason about cryptography that well. I asked a human cryptographer and then pointed out the same flaw but it took them slightly longer.

To me it felt like knowledge creation, but maybe someone had already published a similar scheme and then someone had published a similar attack and ChatGPT was pattern matching on that and adapting it to my setting. That seems very likely because while I was working in RSA, the fix to the attack is very similar to the use of a nonce in Schnorr signatures and for the same reason.

This sounds very interesting! If you ever figure out how it came to that conclusion, that would deserve some write up.

Maybe I will I'll blog about it. You are the second person to ask for a write up.

Knowledge creation doesn’t seem hard enough. I think an a/b testing service that optimizes based on how users respond is doing science, and knowledge creation.

I feel like compressing knowledge and being able to answer questions about the knowledge, with the ability to explore “what if” differences seems more like it suggests real understanding. And that’s what these LLMs can do.