Agreed. I chatted with it about the Chinese remainder theorem today and it gave me an example that didn't work and then insisted that 30 is not divisible by 2 when I questioned it. This was simple enough to spot but I was halfway through the example by the time I realized I couldn't trust it at all. Its confidence is annoying.

It told me the longest known prime number ended with a 2. Definitely not accurate when it comes to math.

ChatGPT is not a calculator, it's a language model.

For me it happend when I asked to write a function using BigQuery, it wrote a function that made a lot of sense but was wrong, because the command didn't exist in BigQuery. When I replay that the function didn't work, it told me, something like this: You right the function that I used it was only working on beta mode, now you have to use the following.... And again it was wrong. I made a little research and never was such a beta commandm.. And then I got that it just makes up things that it don't know, but says it with authority.

I asked it to write a function in Python that would return the list of AWS accounts in an organization with a given tag key and value.

The code looked right, initialized boto3 correctly and called a function on it get_account_numbers_by_tag on the organizations object.

I wondered why I never heard of that function and nor did I find it when searching. Turns out, there is no such function.

It gives the old saying "The reasonable man adapts himself to the world; the unreasonable man adapts the world to himself; therefore all progress depends on the unreasonable man." a new twist, doesn't it?

1. AN AI MODEL IS GIVEN ENOUGH CAPACITY to capture (some of) our human perspective, a snapshot of our world as reflected in its training data. <== We've been here for a while

2. AN AI MODEL IS GIVEN ENOUGH CAPACITY to fabulate and imagine things. <== We're unambiguously here now

The fabulations are of a charmingly naive "predict the most probable next token" sort for now, with chatGPT. But even as a future model is (inevitably) given the ability to probe and correct its errors, the initial direction of its fabulations will still reflect that "inception worldview" snapshot.

For example, if a particular fashion trend or political view was popular around the time the model was trained (with training data typically skewing toward the "recent", simply because "recent" is when most digital data will have been produced), that model can be expected to fabulate along the lines of that imprinted political view.

3. AN AI MODEL IS GIVEN ENOUGH CAPACITY to make the is-vs-ought choice between "CORRECT ITSELF" = adapt to the world; or "CORRECT THE WORLD" = imprint its worldview back onto the world (probably indirectly through humans paying attention to its outputs and acting as actuators, but that makes no difference). <== We're getting there rapidly

Will it be more reasonable or unreasonable?

And which mode wins out long-term, be more energy efficient in that entropic struggle for survival that all physical systems go through?

I am not sure if this is AI generated or meant to read like it is. But I’ll bite.

One thing I noticed, it’s either trained naturally or tweaked by humans not to be political or say anything controversial. I asked if a simple question “Does open door have a good business model”. It punted like any good politician.

It sounds a lot similar to normal thinking errors that we make.

The second time, it gave me code that was almost right.

Just now I asked

Write a Python script that returns all of the accounts in an AWS organization with a given tag where the user specifies the tag key and value using command line arguments

I thought the code had to be wrong because it used concepts I had never heard of. This time it used the resource group API.

I have never heard of the API. But it does exist. I also couldn’t find sample code on the internet that did anything similar. But from looking at the documentation it should work. I learned something new today.

BTW, for context when I claimed to be a “subject matter expert” above, I work at AWS in Professional Services, code most days using the AWS API and I would have never thought of the solution it gave me.

I just asked it to provide a specific example of a security breach caused by a Java serialization flaw and it just made something up, i.e. it found some random security breach (unrelated to anything Java) and found some other random Java serialization bug and claimed that breach was due to that bug. A few minutes on Google revealed they were completely unrelated. Ouch.

It seems like if the contextual analogy here was carried through and ChatGPT were to leave its message as a comment on a Java-security thread, the same approach would apply (a few minutes of research on Google) and the provably-incorrect message would either be downvoted or commented on, just like a human comment with the same content would be.

My biggest fear for the short term is that tools like ChatGPT would allow spamming most of the internet with the equivalent of a Gish Gallop -- so much plausible-looking bullshit spewed out in a short time that it would be a lost cause to attempt to sort through it.

If content known to be of human origin could be archived and frozen now in late 2022 it may become valuable in a few years. Some kind of verifiably timestamped encryption might be useful.

i’ve seen so many examples of this over the past few weeks. just thinking about how many people will just eat what it feeds them borders on terrifying.

in so many instances, it’s just wrong but continues on so confidently.

one of the things i find most interesting is that it has no idea when it’s wrong so it just keeps going. we already have a fairly significant growing problem of people who refuse to admit (even to themselves) what they don’t know, and now this will just exacerbate the problem.

it’s like the worst of debateBro culture has just been automated.

> in so many instances, it’s just wrong but continues on so confidently

Sounds sociopathic, and also like many politicians and people in leadership positions.

> The problem with ChatGPT is that it often reads authoritative. But is often just flat out wrong.

That sounds pretty damn human to me.

yeah, tough to distinguish human BS, from an incorrect ChatGPT.

I guess after cookie banners this is going to be the next frontier of regulation: prohibiting AIs to lie to humans.