As a result, it can't replace a real human being backed by real experience and thoughts, hence it can't be as useful. I converse with myself, too, but it's not the same either with a different human being or an artificial construct which can babble meaningful sentences.

Currently, we simply ignore or don't understand the fact that the meaning of a sentence is not solely built with word order. There are higher orders of information carried implicitly, and they are not articulated in words. Hence an LLM can not replace, even mimic a real human being in a conversation.

Humans for sure do not guarantee this either.

I think of LLMs as an amazing rubber duck. It's heard of everything, and it always responds with something that sounds like it came from the same sphere. You have to use your own mind to figure out if it's meaningful, but this is not so different from conversing with a person. People can babble too.

> There are higher orders of information carried implicitly

You're not always looking for those in a conversation. Sometimes you really are just checking that you've thought things through. Like if someone asks you what the arguments in favor of democracy are, you want a list of points so you can check you haven't forgotten something.

That was my first thought too.

https://en.wikipedia.org/wiki/Rubber_duck_debugging

But it's a pretty amazing rubber duck that has read the entire internet and thus can correlate your ideas with the ideas of all of humanity in a split second on the spot.

However, humans are more nuanced than that. One might remember wrongly, or act in bad-faith. This is why I said "baseline honesty". One people's traits definitely affects how their words are perceived, and this is not carried in the sentence itself.

> I think of LLMs as an amazing rubber duck. ... People can babble too. (Snipped for brevity)

Humans are not as random as an LLM.

> You're not always looking for those in a conversation. Sometimes you really are just checking that you've thought things through.

Again the person you're asking or answering has their own character and their or your words are affected by that implicit knowledge. This is a background process we're not aware of unless you dig into yourself and look for it.

Humans you know and trust, sure. Humans you think you're conversing with online, not so much.

Two decades of experience shows me otherwise. Even your two comments shows a consistent tone. We're just started to discuss, yet I have started to build an image of you thanks to your comments.

This is one of the mechanisms we don't fully understand and don't dare to dig much, because tinkering with people's minds are dangerous.

Many people argue convincingly that it's exactly how the meaning is built, though.

Or, if word order built the meaning and it's devoid of the character itself, we wouldn't be praising authors for their character building skills and embedding things not written or spoken into their stories, regardless of the medium it's presented.

"Honesty" and other forms of self-introspection is just a high level construct which current models aren't trained for, and likely not a fundamental issue with being a stochastic parrot.

However, I'm still on industry forums and communities like HN to hopefully have new and conflicting thoughts thrown my way by peers. That's my primary concern with LLMs, a lack of fresh perspective that has all the nuance of experience and understanding behind its output.

It made a few errors in at least presenting my ideas, but these errors were a consequence of my misunderstanding and lack of clarity.

The errors themselves, in my opinion, are invaluable because they force you to think, to be clear, and to guide the model into giving me useful ideas to look into.

For all intents and purposes, it was an example of the Socratic method, albeit inverted, where the student is asking questions to an all knowing teacher, and the teacher / LLM responds with ideas and hints. Ultimately, it's up to the student to synthesize the solution, be critical of the data, and tie everything together.

No, the brain does not "replace written information with a pointer". I cannot tell you with complete confidence whether writing stuff down improves recall or diminishes it, but I can tell you that you can construct computer analogies in support of either case.

I could posit that writing actually improves your memory, because the minute movements of your arm require more of your brain to compute, therefore creating more neural connections and possibilities of recall.

This is likely just as wrong, but demonstrates that thinking up some vague explanation is insufficient to make a point about a system as complex as the human brain.

In all cases I think parent post is trying to communicate that they are putting the large mass of information elsewhere, and only retain a reference locally. (implementation details notwithstanding)

No, because I literally don't remember my to do list after writing to my EDC notebook. The only thing I know is, it's in the notebook. So I only know the address of my to do list. I'm a literal sitting duck if I leave my notebook at home.

> I cannot tell you with complete confidence whether writing stuff down improves recall or diminishes it ...

I can tell you because I observe myself over 5+ years since I started to write my daily plans consistently to my notebook.

> I could posit that writing actually improves your memory, because the minute movements of your arm require more of your brain to compute, therefore creating more neural connections and possibilities of recall.

What it improves is not the "verbatim memory", but the concepts you work on. Working on something slowly, and with a reflection medium (i.e. pen & paper) allows your brain to refine the thing you're working on better and store it as a concept, rather than memorized string.

> This is likely just as wrong, but demonstrates that thinking up some vague explanation is insufficient to make a point about a system as complex as the human brain.

I have read a paper about supporting the claims I did in this comment, but I failed to find it in a whim. Will update/reply this comment if I find it. I use these analogies because of that paper.

[1] "Michael Levin - Plasticity without genetic change: bioelectric embryos & synthetic proto-organisms", https://youtu.be/5ChRM4CEWyg?t=1169

Why? Those are perfectly good parallels, and quite specific at that. Forgetting the thing, and replacing that memory with the memory of the location of that thing - that's an "indirect reference", i.e. a pointer in programming terms.

> I cannot tell you with complete confidence whether writing stuff down improves recall or diminishes it, but I can tell you that you can construct computer analogies in support of either case.

You can construct analogies to anything in support of anything, what matters is whether the analogy is bringing the reader towards or away from the truth.

Also, even when you have two similar analogies pointing in opposite direction, that doesn't mean either is wrong - they may be talking about two different things. For instance:

> I could posit that writing actually improves your memory, because the minute movements of your arm require more of your brain to compute, therefore creating more neural connections and possibilities of recall.

I believe that to be true just as much as GP's pointer analogy. In fact, I attribute my learning of irregular verbs in English directly to my grandfather insisting I copy the verb table by hand several times. Doing it, I went from near-zero recall to near-perfect recall in two sessions (with each having me copy the table once or twice).

Now the difference here is, in case of irregular verbs, I wanted to memorize them. The act of writing the information down myself somehow helped commit it to memory, perhaps because the brain was more focused, or because I was more invested, or a bit of both.

In case of GP's (and mine too!) experience with TODO list, the act of writing serves the explicit purpose of getting the thing out of your head. This is the opposite of memorization - you literally want to forget about tasks you write down, so they stop circling around in your head, distracting and stressing you.

Incidentally, when I write some TODOs by hand, I actually improve my immediate recall of them - should I try and check. But I don't, because I don't want to reinforce those thoughts. My brain understands, and before I know it, I no longer remember most of them in detail.

And, unfortunately, as you get older and memory starts to fill up, the pointers get overwritten.