>So I asked ChatGPT to explain it to me, I came up with my own mental model from it's explanation, told it that, then it corrected me where I misunderstood things. We went backwards and forwards for an hour, me asking questions, it correcting me, until I felt like I understood the whole picture.

If you're only checking your understanding against the one source you used to obtain it, how can you tell whether your understanding coincides with reality (or rather, with general scientific understanding), and not just with the source you read? And I'm not asking just about ChatGPT; the same question could apply to any source. Books are not exempt from containing errors.

Self-consistency and consistency with your lived experience are good heuristics. Reality is self-consistent, so anything that doesn't add up indicated an error in the source or your understanding of it (or both).

EDIT:

I think it's a kind thing you need to tune yourself into. OTOH, I've observed many (most?) people seemingly being completely oblivious to self-consistency issues of their beliefs and mental models, or even texts they're reading or instructions they're following, and yet... somehow they're generally more successful at life because of it ¯\_(ツ)_/¯.

How do you do that with topics not directly experimenceable, like quantum mechanics or, as in this case, silicon manufacture?

Not the OP, but I don't think you can. The understanding someone manufacturing silicon has will be not be attainable by reading about it etc. Similar for QM, no replacement for doing (some of) the math yourself, be involved in experiments etc.

The very first thing I mention: self-consistency. It's the only thing you have if you don't have any empirical data. It's the only thing anyone has, really. QM scientists reading QM papers and experiment reports of other people, and talking with each other, are still relying on self-consistency to sniff their own (or other people's) mistakes.

>The very first thing I mention: self-consistency.

So if I consistently tell you that that lithium atoms are heavier than carbon atoms, that would make it more likely to be true?

>QM scientists reading QM papers and experiment reports of other people, and talking with each other, are still relying on self-consistency to sniff their own (or other people's) mistakes.

Physicists don't need self-consistency. They can test consistency against reality itself. That's not self-consistency, that's just plain old empiricism.

> So if I consistently tell you that that lithium atoms are heavier than carbon atoms, that would make it more likely to be true?

Nope. But if you said that, and used it as part of an explanation of some process, and every step logically checked out, and the outcome checked out too, and agreed with other things you said, and other things others said, then yes, I'd be likely to believe you.

Alas, a quick look at the periodic table raises a red flag - your explanation is inconsistent with the periodic table and what I know about its structure.

That doesn't necessarily say you are wrong - could be me. But judging by the tone of your comment vs. heaps of other things I know that are consistent with my understanding of chemistry and inconsistent with your statement...

> Physicists don't need self-consistency. They can test consistency against reality itself. That's not self-consistency, that's just plain old empiricism.

No they can't. No one has that much time or money. Physicists aren't routinely replicating every core result empirically for themselves. They rely on the descriptions of experiments and data that they read, and the self-consistency and extreme interconnectedness of reality, which means that wrong information will not add up with someone's experiment, expectations, or lived experience somewhere, and will be quickly flagged as wrong.

Empiricism is only useful because reality is self-consistent. If it weren't, you couldn't really infer anything from empirical evidence because things would just be whatever they wanted to be.

(Or more precisely: we can't prove reality is self-consistent, but if it isn't, nothing can ever make any sense, and we may just as well pack up our technological civilization and go back to the caves we crawled out of. Fortunately, empirical evidence supports the notion of reality being self-consistent to the extent we can observe it :).)

>But if you said that, and [...] I'd be likely to believe you.

That wasn't what I asked. I was very deliberate, I asked if it would be more likely to be true. What's under discussion is not your standard of evidence, but whether non-contradiction by itself is sufficient to conclude that a claim is true.

>Alas, a quick look at the periodic table raises a red flag

That's not self-consistency anymore, that's cross-corroboration. Which, yeah, good on you if you do that, but it's not the process being proposed either by the OP or by fy20.

>Physicists aren't routinely replicating every core result empirically for themselves.

OK, but that wasn't what I said. A physicist can't test every result, sure, but he can test those that are most relevant to his work. I'm not going to get into the philosophy of empiricism because it's not relevant here. My point was that an expert reading a peer's paper is not the least bit comparable to a layman reading an LLM's summary of a field of study. They're just not similar situations. One has the context and the capability to detect bullshit, while the other does not.

> consistency with your lived experience

Aka confirmation bias.

We like explanations that fit what we expect, even if they're completely wrong.

I said consistent with your experience, not with your expectations. Lived experience is empirical data.

That would be the case if our memory wouldn't be completely fluid and built on subjective perceptions. The two sort of merge together over time, you come to see what you expect. It's only when something really out there breaches the belief model when that actually happens.

Reality is most likely self-consistent, but as we can only experience a tiny part of it it‘s impossible to tell. Also, parts of reality may appear to be contradictory with each other when some pieces are not known.

> Also, parts of reality may appear to be contradictory with each other when some pieces are not known.

Exactly. This tells you where something is off. The problem may be your lack of understanding or wrong understanding, or it may be with the source, or the framing, or you may have hit a genuine lack of data - still, the puzzles don't fit in some area.

And yes, not all self-consistent understanding is correct. But all inconsistent understanding is incorrect. And the more knowledge you gain, the less likely it is that it'll all connect self-consistently, but still be very wrong.

> But all inconsistent understanding is incorrect.

Why do you believe that?

Even if we assume that reality itself is self-consistent (what does that even mean?), why would that imply that we humans are able to find a self-consistent representation of it? Maybe reality is self-consistent in some sense but cannot even be represented by the tools we use for theory building.

My point being, the ultimate target of our understanding may be self-consistent, but the way we _necessarily_ have to reduce it to lossy theories means that we can only ever approach it with a non-zero error. And a theory focusing on one aspect, minimizing representational error from one direction of approaching it, necessarily has to make assumptions that will contradict those made by another theory trying to minimize representational error coming from another direction / domain.

You don't seem to be disagreeing with GP, you're just saying that correctness may be practically impossible.

> but the way we _necessarily_ have to reduce it to lossy theories means that we can only ever approach it with a non-zero error.

Yes. Fortunately, we also usually don't need the error to be zero. In practice, we usually have narrow scope at any given time, and can get away with a lot of error.

E.g. people in the past found alternatives to modern germ theory, involving evil spirits and other such shenanigans, but to the extend they covered the high-level mechanisms (curse transfers through contact, hygiene and boiling water removes the harmful effects, etc.), it doesn't really matter the theories were wrong. The beliefs were consistent with each other and empirical evidence to some degree, and to that degree were useful.

> And a theory focusing on one aspect, minimizing representational error from one direction of approaching it, necessarily has to make assumptions that will contradict those made by another theory trying to minimize representational error coming from another direction / domain.

That's fair. Our brains and attentions are finite, you always have to limit the scope. If you imagine you'd have practically forever, you could sort it out and make it all consistent to arbitrary degree (subject to fundamental computational limits, which are physical limits). In practice, the heuristic of consistency works like this:

- For things within your domain of interest, inconsistent information flags an error.

- For things at the interface between your domain of interest and another domain(s), inconsistent information flags an abstraction boundary. It's where you can observe simplifications both domains make because they don't add up (and if you adjust them to make them mutually consistent, you just allowed two domains to work together).

> I've observed many (most?) people seemingly being completely oblivious to self-consistency issues of their beliefs and mental models, or even texts they're reading or instructions they're following, and yet... somehow they're generally more successful at life because of it

What do you mean by "successful at life" here? Genuine happiness, fulfillment in life? Or in the sense of doing well by what society holds as it's current interpretion of what one should strive for, and otherwise just kinda drifting through life?

Because if it's the latter, I'd say that is to be expected. It's much simpler to put your energy into fulfilling the expectations of whoever is your superior in your current group, mostly get the expected reward, and then just coast. Reflection and experimentation, which is required to get to self-consistent views, takes effort and and the willingness to question existing beliefs, which will also be uncomfortable times.

Same question applies if you only check it against your children's biology books

Uh, yeah. That's what I said. Did you have a point or...?

For majority of people, especially children, book is an authoritative source. Virtually no one outside of research fields and very excited enthusiasts will check sources in depth to see whether what is shown in the book is true or not.

So, I think the point OP is making is that most people don't really check sources while learning things 'the conventional way'.

We have to distinguish learning from, let's call it, recreational education. If you're learning a topic, you're not going to be using a single book. Not even schoolchildren do that. Regardless of whether you consider a book an authoritative source, if you read multiple books those books will have to agree with each other, and where they don't you'll experience some confusion until you can resolve the contradiction.

If you're only ingesting information from a single source, be that a book, a teacher, or an LLM, then you haven't really learned, because your knowledge base is unmoored. You can't learn history by studying Tolkien's mythos.

How can you really say you you've learnt the alphabet if you haven't read the document in which the first use of the letter Y appeared?

Because you can successfully use it to communicate with other people. That's the grounding truth. If you'd only ever read about it and never actually employed it then yes, you would not be able to make that claim.

That's exactly what OP was doing, or does teaching kids not qualify somehow?

It's not the same thing. A language is a closed system, it just requires that the participants agree on the rules in order to be successful. Biology is not a language, it's a field of study of phenomena that exist independently of minds. If you read a biology textbook (or an LLM's output about biology) and then relate to someone else what you've read, you and the other person can discuss the topic, and be no closer to ascertain whether what you two understand has any basis in reality.

> Biology is not a language, it's a field of study of phenomena that exist independently of minds.

Your personal assertion is quite wrong at a fundamental level. Biology refers to the field or study, not individual specimens. The field is comprised of the understanding that people over time compiled on nature, along with arbitrary frameworks that help people organize and reason about the topic.

Take the concept of species, and species classification. A specimens exists regardless of being classified or not. However, the same specimen can be classified differently depending on the state of the body of knowledge at the time. In some cases you had species being reclassified due to new findings, such as genetic tests.

Once you understand this, you learn that you can't claim that a field of study is anything other than abstractions and partial and incomplete observations compiled form people throughout time.

Biology is in some sense a language, how do we even agree on what the boundaries of biology vs other domains of science are without participants agreeing on the rules?

In any case I think you have an overly narrow definition of learning that we're unlikely to come to terms over.

>how do we even agree on what the boundaries of biology vs other domains of science are without participants agreeing on the rules?

Those boundaries are completely imaginary and don't exist in reality. In reality there's no biology, there's only elementary particles interacting physically. Whether you agree with someone else to classify a phenomenon as biological or chemical, you're not refining your knowledge of the real world, you're just performing an organizational task.

But, say, how many times the tympanic ear evolved independently is a real phenomenon that can be investigated, and it's something that you're either correct, incorrect, or ignorant about. If I tell you it evolved five times, what more can either of us gleam about the real facts by just discussing this factoid back and forth, if neither of us has access to any additional knowledge or way to put this datum to the test?

He's talking about learning that makes you operational. If you can speak with someone, then you've learned language. If you can get a computer to do something, then you've learned some programming.

The equivalent for biology would be to grow a plant or a few plants and animals successfully. That's operational at a certain level, you could also be operational at a lower or higher level.

No, it doesn't necessarily need to be applicable, it just needs to be testable. The size of the Earth for example may not affect you directly in any way, but it's something you can verify to some degree of precision. But if you just read about it, can you really say that you know it? What if all you believe about the size and shape of the Earth is from what you've asked an LLM? Do you still know it?

EDIT: Perhaps not the best example, because the size and shape of the Earth are data that are repeated often enough that an LLM would be unlikely to quote it grossly incorrectly, but I think my point still comes across.

I’ve been burned by this enough that I no longer say that I’ve learned something if my only interaction with it is explanation from books. You can get snippets of knowledge and a framework of understanding, but true learning only comes with deep interaction of the concepts (practice, simulation, experiments, observations) and not merely reading.

Concrete high-profile example: in "Surely You're Joking, Mr. Feynman!", Feynman told of a ball which, in a Brazilian college-level physics book, was described as having a 40% higher acceleration that it would actually have in practice.

Turns out that the author had done a thought experiment but neglected to factor in the rotational inertia.

Or do you? Cause if you went back and forth with ChatGPT for an hour it definitely hallucinated and lied to you at some point. Maybe consider using something else like Brilliant.org if you want to learn a topic, yanno, so you don’t propagate whatever hallucination from ChatGPT to your kids.

These days online courses like Brilliant and such are likely largely LLM generated. Are they actually vetted by experts before publication? Who knows. My money is on no, or at least, not until someone complains.

Its easy enough to prompt ChatGPT for primary sources when doing research to validate any claims its making.

You should actually do this. Ask ChatGPT to check what it said against sources. It does correct itself. The corrections are usually not major. Just mental shortcuts.

I personally do, but something tells me the commenter i was responding to wouldn't trust the llm correcting itself.

This would be a valid point maybe 3 years ago, but most chatbots will now query and verify direct sources, especially in research mode.

This is very simple to validate and verify. You could argue it may find false primary sources.

You can condemn models for a variety of other things, but acting as if this is still reality shows a lack of understanding as to modern model capabilities

Your comment is phrased as if it somehow refutes their point but it doesn't.

> Cause if you went back and forth with ChatGPT for an hour it definitely hallucinated and lied to you at some point.

If you're asserting that this is not the case today then that's going to be require pretty extraordinary evidence. "Chatbots use Google now" is not evidence that the information they provide is in fact correct.

They don't hallucinate all the time like they used to, no, but I'd be very surprised if the majority of these sorts of conversations were free of major factual errors.

I frequently notice degradation in the model model's ability to remain coherent when it searches for information online. For example I might ask Sonnet 5 "how do I build a shed" and during its search it presumably comes across an article which talks about building a shed out of paper mache, then the model responds with something like "I caution you against your plan to build a shed out of paper mache" -- Wait, what? Who said anything about building it out of paper mache?

Firstly, I can't refute a point that's little more than an opinion - my doubt is that a modern frontier model is significantly hallucinating within a relatively short conversation that can easily be verified. There is no way to refute or accept the point without a complete conversation log. I am criticizing the hyperbolic nature of the comment.

I doubt you are getting to the context level of model degredation where it reaches context limits within a verbal hour conversation.

I've just tried to recreate your example on sonnet 5, and as someone who has done DIY projects it reads completely appropriate, but I'm happy for criticism from a shed builder. It never once tells me about paper machie or creates a silly example.

This is via a prompt requesting tools and materials, and could be further improved, unfortunately, I can't paste the markdown formatting provided.

""" Reference size used below: 8x10 ft shed. Scale material quantities to your dimensions.

Step 1: Check Regulations & Plan Materials: None yet — just your design/plan (graph paper or free shed-plan software)

Tools: None

Skills to find: None required, but if your shed is large or near a boundary, a quick chat with your local planning/building department saves headaches later

What to do: Confirm permit requirements, setback distances from boundaries, and max height/size allowed without permission. Sketch your design and finalize dimensions.

Step 2: Prepare the Site Materials: Landscape fabric (weed barrier) Gravel/crushed stone (for drainage base, ~4-6 in depth) Marking spray paint or stakes + string

Tools: Shovel & spade Wheelbarrow Rake Hand tamper or plate compactor Spirit level (4 ft) or laser level Tape measure Builder's square (for squaring corners)

Skills to find: Basic site leveling — not hard, but a laser level rental helps a lot if the ground has any slope If you have poor drainage/heavy clay soil, worth asking a landscaper for advice What to do: Clear vegetation, mark the footprint, excavate and level, add compacted gravel base for drainage.

... """

I won't include the whole document, can share it further but anyone can replicate just by asking sonnet

I just don't understand the need for such hyperbole, and pretending that models are still gpt3, when you can get counter evidence in seconds.

It reminds me of the craze teachers had against trusting Wikipedia - yes, you shouldn't take all claims at face value, but arguing that nothing from Wikipedia could be useful just makes the argument silly.

Err, that wasn't meant as literal example because real conversations obviously have more than 1 turn. The only thing you could prove by getting a different result is that they don't _always_ do that, even if I shared the full conversation log.

You "don't understand the need for such hyperboles" because they're not hyperboles, I don't know how you can not pick up on these errors in your own conversations.

> and pretending that models are still gpt3

I explicitly said that the new ones are better. How's that for hyperbole?

Then this entire conversation is a pointless argument - as I agree that models aren't omniscient, godlike entities that are perfect sources of truth, and that you need to use critical thinking when using them.

I don't trust models blindly, and interrogate and verify claims that they make, but that's a basic component of being a human being.

I also never try to have massive multi step conversations to the point where I'm nearing the context limits, as if there's a subclaim i need to interrogate it's far better to clear context and just start a new chat, I have notes to join up ideas.

When learning, I'm not just doing so blindly asking a model questions, I have other material up, I can look at the answer to a example question from a textbook to verify whether I have used a model to successfully learn.

This conversation is just going to devolve further into a "well it doesn't always work" to which yes, I agree, but that doesn't mean it's not useful and doesn't help the learning process.

I would trust ChatGPT more than the average elementary school biology teacher.

Can you say in all honesty that a read of the Wikipedia article for DNA would have been less helpful? It's less convenient perhaps, but definitely more authoritative.

Wikipedia is a great _reference_ but not necessarily the best way to learn about a topic. Of course, this depends on the topic, on who has been writing the page, etc

A particularly bad example is higher maths - a wiki pages on a complex mathematics topic often reads like "A gruncheon is a worch in the brashation of plusters" and each of these words is a separate page or topic. Of course, you _can_ in theory 'just' click through all the tree of linked pages to understand a concept ...

For DNA the page (scanning it now) is well laid out, with images (including a spinning Rasmol? image) and lots of detail. However, the detail could be a drag on understanding for some : There are 'nucleotides' and 'nucleosides' and 'nucleobases'? There are non-canonical bases? Supercoiling? Z-DNA? While I know (most) of these things, it is because I've learned about them in other contexts, or by direct instruction.

I'm not saying it is impossible to understand DNA from that page, but it is likely to be harder (for some?) than a more conversational approach to learning.

If only Wikipedia contained like an easier-to-digest version of Wikipedia, using planer language: https://simple.wikipedia.org/wiki/DNA

I just asked Opus to "explain DNA to me in simple language" and the two are not even in the same league in terms of quality.

Fair point - but I wonder how many people that have visited Wikipedia know about simple wiki. I was probably aware of it, but not enough to remember to suggest it as an alternative!

I mean - it is certainly better ... but it is still a lot of stuff. For example:

> Part of an organism's DNA is "non-coding DNA" sequences. They do not code for protein sequences. Some noncoding DNA is transcribed into non-coding RNA molecules, such as transfer RNA, ribosomal RNA, and regulatory RNAs.

Do you _need_ to know about tRNA, rRNA, and operons (?) to understand DNA? The thing about an encyclopedia/wiki entry is that it has to cover the whole topic. This is a strength for reference, as you can scan it and find the bit you need. For learning from scratch, I can see that a conversational approach (with a human or LLM) has advantages where the learner can direct the level of detail and path through the material.

Ultimately, both are worthwhile, but I can also see the strengths/weaknesses of both ways to learn.

Just reading something is a fundamentally different experience that being actively involved in a conversation about the thing.

One thing you can do with the LLMs is ask them to make up quick quizzes for you one a subject.

That can help a bit.

Wikipedia is completely non-interactive so of course the experience is different. Do you want to look up something like https://en.wikipedia.org/wiki/Principal_component_analysis and try and understand that from Wikipedia? If you don't understand it, you have to try and click elsewhere. With an LLM, I can be very specific, "I understand x, y, z about PCA but I don't understand why we have to do it? What happens if I don't apply it?" and the LLM most of the time will give very approachable explanations that can be refined further if I still don't get it.

I am studying for a Masters degree in Computer Science with AI and the lecture notes are like Wikipedia sometimes. Incomplete, perhaps assume pre-knowledge that lots of Masters students won't have. All of these I have taken to ChatGPT and got great explanations, diagrams, graphs etc.