>that it freely believes contradictory facts without being confused,

Humans do this. You do this. I guess you don't have a meaningful world model.

>freely confabulates without having brain damage

Humans do this

>and it has no real understanding of quantity or causality.

Well this one is just wrong.

>That's not much of an excuse, computers exist because they do the things humans can't do very well like following long repetitive lists of instructions.

Computers exist because they are useful, nothing more and nothing less. If they were useful in a completely different way, they would still exist and be used.

The stuff that comes easy to us, like navigating 3D space, was trained by billions of years of evolution. The hard stuff, like language and calculus, is new stuff we've only recently become capable of, seemingly by evolutionary accident, and aren't very naturally good at. We need rigorous academic training at it that's rarely very successful (there's only so many people with the random brain creases to be a von Neumann or Einstein), so we're impressed by it.

>Well this one is just wrong.

Is it?

--

Me: how many characters are in: https://google.com

ChatGPT: The URL "https://google.com" has 12 characters, including the letters, dots, and slashes.

--

What is it counting there? 12 is wrong no matter how you dice that up.

Part of the reason is it has no concept of the actual string. That URL breaks into four different tokens in 3.5 and 4: "http", "://", "google" and ".com".

Its not able to figure out the total length, or even the length of its parts and add them together.

I ask it to double check, it tells me 13 and then 14. I tell it the answer and suddenly its able...

---

Me: I think its 18

ChatGPT: Let's recount together:

"https://" has 8 characters. "google" has 6 characters. ".com" has 4 characters. Adding these up gives a total of 8 + 6 + 4 = 18 characters. You're correct! My apologies for the oversight earlier.

---

Count me out.

Humans process photons, but only after converting them into nerve impulses via photoreceptor cells in the retina, which are sensitive to wavelengths ranges described as "red", "green" or "blue".

As a result, humans are not very good at distinguishing different spectra that happen to result in the same nerve impulses. That information was lost by the conversion from photons to nerve impulses. Sensors like the AS7341 that have more than 3 color channels are much better at this task.

I am a strong proponent of LLM's, but I just don't agree with the personification and trust we put into its responses.

Everyone in this thread is defending that ChatGPT can't count for _reasons_ and how its okay, but... how can we trust this? Is this the sane world we live in?

"The AGI can't count letters in a sentence, but any day not he singularity will happen, the AI will escape and take over the world."

I do like to use it for opinion related questions. I have a specific taste in movies and TV shows and by just listing what I like and going back and forth about my reasons for liking or not liking it's suggestions, I've been able to find a lot of gems I would have never heard of before.

I suspect we might find that adding in the multimodal visual and audio aspects to the model gives these models a much better basis for mental arithmetic and counting.

Most LLMs can just pass the string to an tool to count it to bypass it's built in limitations.

Does a blind man not understand quantity because you asked him how many apples are in front of him and he failed ?

I don't think that test determines his understanding of quantity at all, he has other senses like touch to determine the correct answer. He doesn't make up a number and then give justification.

GPT was presented with everything it needed to answer the question.

>he has other senses like touch to determine the correct answer

And? In my hypothetical, you're not allowing him to use touch.

>I don't think that test determines his understanding of quantity at all

Obviously

>GPT was presented with everything it needed to answer the question.

No, it was not.

The deaf example now is like asking GPT "What am I pointing at?"

Imagine a legally blind person, they can barely see anything; just general shapes flowing into one another. In front of them is a table onto which you place a number of objects. The objects are close together and small enough such that they merge into one blurred shape for our test person.

Now when you ask the person how many objects are on the table, they won't be able to tell you! But why would that be? After all, all the information is available to them! The photons emitted from the objects hit the retina of the person, the person has a visual interface and they were given all the visual information they need!

Information lies within differentiation, and if the granularity you require is higher than the granularity of your interface, then it won't matter whether or not the information is technically present; you won't be able to access it.

Also the blind person wouldn't confidently answer. A simple "the objects blur together" would be a good answer. I had ChatGPT telling me 5 different answers back to back above.

If the legally blind person never had had good vision or corrective instruments, had never been told that their vision is compromised and had no other avenue (like touch) to disambiguate and learn, then they would tell you the same thing ChatGPT told you. "The objects blur together" implies that there is already an understanding of the objects being separate present.

You can even see this in yourself. If you did not get an education in physics and were asked to describe of how many things a steel cube is made up, you wouldn't answer that you can't tell. You would just say one, because you don't even know that atoms are a thing.

ChatGPT can't count, the problem is the tokenizer.

I do find it funny we're trying to chat with an AI that is "equivalent to a legally blind person with no correction"

> You would just say one, because you don't even know that atoms are a thing.

My point also. I wouldnt start guessing "10" and then "11" and then "12" when asked to double check only to capitulate when told the correct answer.

First of all, it obviously changes everything. A shortsighted person requires prescription glasses, someone that is fundamentally unable to count is incurable from our perspective. LLMs could do all of these things if we either solve tokenization or simply adapt the tokenizer to relevant tasks. This is already being done for program code, it's just that aside from gotcha arguments, nobody really cares about letter counting that much.

Secondly, the analogy was meant to convey that the intelligence of a system is not at all related to the problems at its interface. No one would say that legally blind people are less insightful or intelligent, they just require you to transform input into representations accounting for their interface problems.

Thirdly, as I thought was obvious, the tokenizer is not a uniform blur. For example, a word like "count" could be tokenized as "c|ount" or " coun|t" (note the space) or ". count" depending on the surrounding context. Each of these versions will have tokens of different lengths, and associated different letter counts. If you've been told that the cube had 10, 11 or 12 trillion constituent parts by various people depending on the random circumstances you've talked to them in, then you would absolutely start guessing through the common answers you've been given.

Except that "http" should be "https". Silly humans, claiming to be intelligent when they can't even tokenize strings correctly.

h t p s : / g o l e . c m

There are 13 unique characters.