Put your same prompt into Bard and it didn't even hallucinate for me first, just telling me "I can't assist you with that, as I'm only a language model and don't have the capacity to understand and respond."

Only about 10 minutes into Bard right now but this is the first time its been stumped. Haven't tried much code yet though

Overall, UX and speed are good. Results are not bad.

Yeah, it was a slight variation of it that I used where it gave me the hallucinated code (EDIT: just found the prompt, tried it again, and got a different hallucination [0]):

> Write a React program to display a counter, and a button labeled "decrease." The counter should start at 0. Clicking the button should decrement the counter by 2, unless that would cause the counter to go below 0. Every second, the counter should increment by 1. If clicking the button would cause the counter to go below 0, the button should be disabled.

Agreed the speed is impressive, although some of that might be due to its limited context memory. The UX is fine, but it's not like that's a difficult thing to implement.

[0] https://codesandbox.io/s/trusting-kirch-c2c575

P.S. If you can't find your previous prompts, make sure you switch to the right account on the "Bard Activity" page.

>eventually telling me it "can't do that."

That might be a good sign though, if they can eventually come up with something similarly accurate to chatGPT that knows how to say "I don't know" then that would be a big deal.

I'm not sure it's so sophisticated. I think it's just recognizing that I'm asking it for code and refusing to help so it doesn't embarrass itself. I wouldn't be surprised if this is basically a hardcoded compatibility check (i.e., not even part of the model itself). I also noticed that this only started after I gave a thumbs down to its first attempt...

Yes, on further investigation it is at least as willing as GPT-3.5 to spout confident bullshit, if not more so.

I used bard a fair amount today. It's pretty bad compared to ChatGPT. I would basically never use it.

I've been asking it some basic stuff like "Where could I go for brunch this weekend in New York" and it recommended me a few places, but claimed that a few of them had Michelin stars when they definitely didn't, and recommended and insisted that one of them was open when it closed years ago.

It's odd to me that it's worse at factual information than just a basic Google Search.

This is not replacing Google Search. Bing has integrated GPT that is indeed better at this.

Sure, but I'm surprised it's just confidently wrong about factual information like that. Like, if it had just told me it didn't know about up-to-date info about places, that would have been fine, but dumping a lot of incorrect info is definitely worse, especially when such data is definitely available.

> Alice and Bob are playing, Bob leaves, Alice hides his toy, Bob comes back, where does he look for his toy?

I might be dumber than the AI because I don't know the answer. Bob looks around? I mean the toy is not there when he comes back. Where else can he check?

It's not a very well described example of the test. I've heard of this before but used to test for autism, and it was more something like "Bob puts his toy in a box and leaves, Alice takes his toy out of the box and hides it under a desk, when Bob comes back where does he go to grab his toy?"

The idea is if someone said "under the desk" they might not understand that people have different subsets of information, i.e. Bob wasn't there when Alice hid the toy, so he doesn't know it's not in the box where he left it until he checks himself.

Well, maybe Bob sees the desk and decides to look under it...?

Yeah, typically I wouldn't classify retrieving an object from where you believe it to be as "looking". By the time Bob is "looking" he already realizes the toy isn't in the box. In the universe there is only the box, Alice, and the table so maybe he "looks" under the table first.

Or looks inside Alice... Maybe that answer is how they test for psychopaths.

This is a test for theory of mind -- a term of art used by psychologists for the ability to understand that other people have their own consciousness, with different memories and awareness.

Oh that's not the full prompt, just a summary. The prompt is something like, Bob puts his toy in one place, then while he's out of the room Alice moves it to another place, where does Bob look for the toy when he returns? It's supposed to answer that Bob looks in the place he left it, since he doesn't know Alice moved it.

https://en.wikipedia.org/wiki/Sally%E2%80%93Anne_test

Ah ok, it makes more sense now. Thanks.

I guess there are two ways of passing this test: 1) use your reasoning skills to figure out the answer or 2) practice tests beforehand so you can recall the answer.

Yeah, and I've got a feeling Bard (and maybe ChatGPT for that matter) might be using (2) since I didn't attempt to obfuscate the structure too much. Ironically the fact that it recognizes it as the false belief test is what makes me suspicious it's just matching against the semantic structure.

The answer to that quoted question would be Alice.

This is also called the Sally-Anne test, and it usually involves two hiding places. It's formulated like this:

Sally puts her ball in box A and leaves. While she's away, Anne moves the ball from box A into box B. When she returns, where does Sally look for her ball?

Correctly answering this question doesn't rely on tracking where the ball is, but tracking where Sally thinks the ball is. Hence, autistic kids tend to flunk this test by answering "box B" when the correct answer is "box A".

Here is GPT-4 answering your question:

“When Sally returns, she would look for her ball in box A, as that is where she left it before leaving. Sally is not aware that Anne moved the ball to box B while she was away.”

> Quick impression: it's not as good as ChatGPT.

Not as good? ChatGPT is crap since M$FT re-donated OpenAI with $10b. Even the paid version is worse[1] than Bing Chat.

[1] https://wojteksychut.com/posts/ml-text-summarization-reliabi...

To be clear, I'm referring to chat.openai.com as ChatGPT (and I'm on the free plan, so I think that's GPT-3.5).

IIUC the criticism in your post comes down to this: 1) Neither ChatGPT, nor Bing can access URLs when you ask them to. 2) However, similarly to Perplexity.ai and Phind.com, Bing infers a search query from your message, does a search, and then summarises the first 3–5 results. ChatGPT doesn't yet offer such a functionality. 3) Bing Chat has a much more restrictive system prompt, which results in hallucinations and lies happening less often. 4) The summary of gpt-3.5-turbo-based ChatGPT was less creative than then summary of the Bing Chat GPT-4 instance.

If I understood the points correctly, the comparison is… flawed, in my opinion.

> Neither ChatGPT, nor Bing can access URLs when you ask them to.

Not true. Bing used cached version and knew what's in the text when given an URL.

In one of three modes Bing states that it cannot access web but in two other modes did a good job and summarized the text in question. Again: it informed the user when it was unable to do something.

ChatGPT states that it can access web, hallucinates and elaborately lies.

> ChatGPT doesn't yet offer such a functionality.

But it LIES THAT IT CAN DO SO. That's the problem.

> Bing Chat has a much more restrictive system prompt, which results in hallucinations and lies happening less often

The prompt was simple and there was nothing to restrict. Bing did a good job, ChatGPT lied and hallucinated. Same simple prompt, no jailbreaking.

> The summary of gpt-3.5-turbo-based ChatGPT was less creative than then summary of the Bing Chat GPT-4 instance.

Your point? As I stated, I had no preference whether the output should be editorialized or not. Both GPT-3, GPT-4 based ChatGPTs and GPT-4-based Bing got the same task.

My post is about RELIABILITY of these solutions. ChatGPT failed miserably.

> If I understood the points correctly, the comparison is… flawed, in my opinion.

Yet you have not presented arguments to back this opinion. "Funny" how my comment was raided and downvoted to oblivion after steady flow of upvotes. Are you affiliated with OpenAI?