GPT 3.5 had trouble understanding when I told it "Say 2 bob are a beb, how many beb per bob are there?" and it wrote a goddamn essay about shoes.

That thing isnt smart, it doesnt understand, it doesnt know, it just rambles. I have worked with people who do the same, yes, but they also werent a threat to most jobs.

I said it before, and I will say it again: If ChatGPT 3,4,5,... can take your job, maybe youre not really providing that much value. Make of that what you will - not everyone has to provide huge value.

Answer from GPT-4:

"This question seems to be intentionally nonsensical or is using unfamiliar terminology. However, if we try to interpret it, we could say that there are 2 "bob" making up 1 "beb." In this case, there would be 0.5 "beb" per "bob." Please provide more context or clarify the terms if you are looking for a different answer."

What do LLaMA-based models answer for this?

alpaca.cpp 7b model gave me this:

> Say 2 bob are a beb, how many beb per bob are there?

There are two bebs per bob.

> No, there are 2 bobs per beb.

Correct! There is only one other answer that makes sense - "1 Bob for every Beb". But this does not make mathematical or logical sense since a single Bob can have multiple Babes associated with it and vice versa (as seen in the example). Therefore, we must conclude there are two bobs per beb.

Cool, but sadly, as I said, it did not give a very useful answer. If asked enough times, im sure it will give a reasonable answer, yes, but thats not the point.

GPT4s answer is interesting, though

But all of the answers were correct and useful, and GPT-4 was perfect. Anyway ChatGPT is getting hooked up to Wolfram Alpha, and that won't have any issues with basic algebra.

If you already know the answer, but need an interesting and detailed way to explain/write it, then GPT4 can be an immense value-add. Even if you have to do some fact checking afterwards.

I typed the query into chat-gpt3.5 (turbo and legacy), and 4, and they all said that there's 0.5 beb per bob.

Did you use the quoted prompt exactly?

No, I didn't use the quoted prompt, but even after explaining to it that bob and beb were not, in fact, shoe related terms, it still kept insisting and being confused (while also giving the correct 1/2 answer).

It can do it, but its not deterministic, and it doesnt really do it well. You can continue the chain by asking "How many bob per bib, assuming two beb per bib?", and see if it chokes then. It sometimes does, sometimes doesnt.

GPT-4:

   If 2 bebs are equal to 1 bib, and we know that 1 beb equals 2 bobs, we can
   determine how many bobs there are per bib using simple substitution.
   
   1 bib = 2 bebs
   1 beb = 2 bobs
   
   Therefore,
   
   1 bib = 2 bebs × 2 bobs/beb = 4 bobs
   
   So, there are 4 bobs per bib.

You do realize that the current implementations get their context polluted by your prior conversation, right?

I think right here we have an example AI reproducibility problem. It seems fully reasonable and credible as an outcome, but it is hard to dig in and replicate. But the truth of ML is it would be difficult to replicate even if things were FOSS.

> I said it before, and I will say it again: If ChatGPT 3,4,5,... can take your job, maybe youre not really providing that much value.

The thing you're missing here is that it might take someone's job not because they weren't providing the value but because the people who crunch numbers decided to axe 15% of a company's workforce because some executive was sold a pack of lies about what LLMs/"AI" are actually capable of.

It's fine if that happens to one company who then finds out the hard way. It's probably more social-unresty if it's essentially done at every company in every marketplace an LLM can touch - from writing to programming to 3D animation to teaching.

The hype machine around LLM/AI here is the same irrational one we saw around blockchain. The key difference is blockchain was basically never sold as really replacing a person's job (at best you could argue it was sold as getting around the banking industry and maybe eventually being able to replace it, ish). The primary sales pitch of these LLMs is essentially "do more with less".