I think of AI like a very eager and bright 20 yr old intern who’s really into whatever tech I’m trying to learn.

Good knowledge of facts, willing to give anything a try, but not much wisdom.

> Good knowledge of facts

I haven’t seen this demonstrated in gpt-4 or Claude sonnet when asking anything beyond the most extreme basics.

I consistently get subtly wrong answers and whenever I ask “oh okay, so it works like this” I always get “Yes! Exactly. You show a deep understanding of…” even though I was wrong based on the wrong info from the LLM.

Useless for knowledge work beyond RAG, it seems.

Search engines that I need to double check are worse than documentation. It’s why so many of us moved beyond stack overflow. Documentation has gotten so good.

ChatGPT and Claude outputs are equivalent to a 20 yr old intern who goes to MIT?

But actually whenever this happens you get rich signal about the hidden didactic assumptions of your method of prompting it about things you yourself are unsure you can verify for yourself, and also of how you thought the tool worked. This is a good meta skill to hone.

In the same way going to a fortune teller can give me rich insight about my love life and work life balance, sure.

A visitor to physicist Niels Bohr's country cottage, noticing a horseshoe hanging on the wall, teased the eminent scientist about this ancient superstition. "Can it be true that you, of all people, believe it will bring you luck?"

"Of course not," replied Bohr, "but I understand it brings you luck whether you believe it or not."

s/luck/lower negative log likelihood. Besides, a participant can still think about and glean truths from their reflections about a conversation they had which contained only false statements.

What point are you trying to make?

That it’s better to be superstitious than not, because maybe it matters?

The internal mechanisms by which a model achieves a low negative log likelihood become irrelevant as it approaches perfect simulation of the true data distribution.

This is how I think about it as well. It’s like the equivalent of trying to build an ML model with only positive examples otherwise

> I have the experience and know how to verify this stuff

That was GP's point. Steve Yegge has also talked about this.

Is that not the opposite point?

Someone who does not have that experience would be much more harmed by the false information LLMs can and do spit out.