This kind of problem is (trivially?) solvable through the ReAct framework, like LangChain etc. Basically you get good data, vector embed it, and make sure the LLM knows where to look for accurate information.

This approach has some issues

- it chunks inputs, with some overlap, but this can destroy context

- the retrieved passages, when they come from different documents, have no apparent relation or could be mistakenly considered related

- the model struggles to correlate data between the document snippets, taking half an idea from one side and half from the other side and mixing them up in something that doesn't really make sense

Implementation details. Check out what's going on with LangChain, augmented retrieval, etc. We'll be able to create knowledge bases on specific subjects with vetted data, and get the bot to retrieve and summarize appropriate results while providing a citation to the original source.

Slight tangent but preferably i would like an AI that can go against the status quo, the elite classes, the mainstream discourses and ignore the circus.

I think one of that most interesting things that can come of bots like GPT-X is that it can make new connections, unravel "stuff", do extremely intricate deductive reasoning.

Be the data driven arbiter of the the truth for everyone, not just the tiny established classes or cultural hegemonies.

The ideological and cultural noise increasingly smokescreen any realpolitical, material or resource-oriented analysis of actual economic power structures in the world in the last years, AI could be a godsend (or the opposite unfortunately).

I remember reading a sci-fi years ago about the stuff an AI concluded when asked philosophical questions that were so bizarre and frightening that people shut it down, and i'm sure we're in the same territory with political and scientific analysis.

It's either dangerous to the orders of the world, or not that interesting and borg like on a philosophical level.

Agreed... An example would be the recent Jordan Peterson mentions in Twitter around the bias in the system. Even without leaning one way or the other politically speaking it should be concerning... because any similar bias can easily be used to target "you".

> Slight tangent but preferably i would like an AI that can go against the status quo, the elite classes, the mainstream discourses and ignore the circus

There's a Charlie Brown comic about this sort of thing. Although I think it's an edit, not an original comic. Something like "They are never going to give you the education you need to overthrow them"

Similarly they are never going to give you an AI that will side with a you against them.

I would also love such an AI though.

Who - WHO - thought that naming something new 'the ReAct framework' in 2023 was a reasonable choice.

For this specific example, an LLM in front of NICE will produce the correct result. It's a matter of time before this case will be fixed.

From a non-expert's perspective through, an LLM is very dependable, unless it completely goes off the rail. How would anyone know when to trust it and when to be skeptical?

Trust but verify. We’ll get processes built around this for accuracy sensitive applications. I imagine it will look something like a GAN configuration with two or more LLMs trained to adversarially critique and fact check the outputs of the other(s). Might even mimic the relationship between the hemispheres of our brains.

Two models trained from different lineages won't hallucinate the same. When you want to check for hallucination the cost is to run the task on two models. For now. But soon it looks like LLMs will be better calibrated. It seems they are well calibrated after pre-training but become less calibrated after fine-tuning and RLHF. The last stage breaks its abilities to estimate confidnce scores correctly.

That's not how LLMs work. Including the NICE, if it actually isn't already, will not guaranty a "correct" result. It will increase the chance that the response is directly coming from the training but there is no guarante. If you are interested in why this is the case you can read this [1] post from Stephen Wolfram on how ChatGPT and in general LLMs work. This might give some insight on how and when to to use it more effectively.

[1] https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-...

Could we instead have the LLM use NICE similar to how Bing uses the web as a reference instead? It still wouldn't guarantee a correct result, but it would that increase the reliability, right?

Could be. That means feedinng it the data from NICE in a prompt rather than relying on the data being in training set. That will intuitively increase the reliability of the answer (I have to look into it a bit more).

On one hand if you already have the NICE data at hand, you already have your answer. There won't be a need for a search enginge or a chat bot other than to perhaps, summerize the data (which is valuable on its own). On the other hand, if you don't have the NICE data at hand, the correctness of the response relies on the accuracy of the search method in order to feed the correct page to LLM. This is an issue additional to LLMs accuracy.

At first it might seem like an easy problem to solve but when one wants to engineer a solution to a nice streamlined product, it's more challenging; unsurprisingly.

This is an interesting comment, and highlights the accuracy problem with LLMs. But this is but one type of query that LLMs are used for, akin to an informational web search (E.g. How tall is the Eiffel tower).

Was it GPT4 or GPT3.5?

It spews less bullshit with each new iteration. Matter of time really.

I don't want to have a calculator (or say bookkeeping software) that gives correct results most of the time but not always, and then hear from the developers that it will get better with each iteration. I need a calculator that is correct 100% of time, not even 99.999%, because otherwise I can't rely on it at all.

In other words, the utility of a calculator that is correct only 99% of time is zero, since you can't even tell when it's wrong.

Imagine a life or death situation where a calculation held the key. Would you rather use a calculator that is right 50% of the time or 99% of the time. The obvious choice highlights that the utility is not exactly zero. But I see your point.

>I don't want to have a calculator (or say bookkeeping software) that gives correct results most of the time but not always

And yet Excel is widely popular, despite several footguns and inaccurate calculations!

Excel has consistent (I guess coherent) flaws/bugs, unlike GPT.

Don’t rely on floating point accuracy!

Those are predictably correct and incorrept, in a very formal way!

So what about large asteroid strike levels of improbability? I have to reject your premise that there is no probability where you can stop worrying about the problem.

(This might be a moot point, because I'm not sure current methods can ever get to this level of accuracy, due to limitations of the training data. Needs an entirely new method or a clever insight to optimize for truthfulness with unclean training data, and InstructGPT hasn't made much progress on this, and it might not even be possible)

Confining an LLM to the very narrow domain of "calculators" is a mistake, I think.

You wouldn't say "a programmer that is 99% correct is worthless, I need 100%". I'm pushing it, but for a more fair comparison I'd say measure it against a programmer. How often are we wrong? 75% of the time? :) being generous here. It's the tools that make us productive.

I don't know about you specifically, but I don't think you'll be very productive with a bare terminal lacking any modern IDE-like or even REPL facilities. I'll ask you to come up with instantly working code every time, all the time. It doesn't work like that. You need iteration and I believe these kinds of AI have the same issues as us. There are wrong sometimes (often) and need feedback.

> You need iteration and I believe these kinds of AI have the same issues as us.

It's funny how we resort to humanizing the machines when their results are inaccurate. We don't do that with the calculator, because it's expected to be 100% bug free. When there's a bug in the calculator code we expect it to be fixed, not gradually improved.

Speaking of bugs: mistakes in code is one thing, wrong output because of a fundamental flaw in the algorithm is another. The statistical machines we are dealing with work as intended, or at least the wrong output the top comment here brings up is not a bug, it's a feature. That's the difference.

That's (IMO) too narrow view of what a "machine" is. Complex machinery of any kind never is 100% correct and needs constant correction and maintenance. I still think approaching this as a "calculator" is awkward at best.

> Complex machinery of any kind never is 100% correct and needs constant correction and maintenance

Computers are extremely close to 100%, we generally expect a CPU to never make errors even after years of working. If it starts making any errors at all we throw it away and make a new one.

>Computers are extremely close to 100%

We must work in extremely different industries!

Do you code in checks to check the calculations made by the CPU? I've never ever seen anyone do that. If a CPU starts making errors we throw it away. A typical CPU will make many quadrillions of correct calculations before its first error, I'd say that is basically 0 errors.

This is a very weird statement that's failing based on logical category.

My computer will pretty much add 1+1 correctly forever never making a mistake.

My computer will perform an 'error' every time I put bad code into it, and some of those logic chains and error conditions are not very obvious.

The issue here is you think the LLM is performing a category 1 error, when the problem we are seeing is a much more human like category 2 error.

Literally LLMs get much better with chain of thought, feedback, and/or consensus.

Gpt-3 performance on MultiArith goes from 18% to 92% with all three. This isn't some hackneyed anthropomizing. Countless research papers showing massive improvement with these processes.

Is the typing the hard part? I’ll look up libraries and apis, pretty regularly. maybe an algorithm every few years.

Figuring out what’s wanted from me takes forever though.

He is comparing it to a calculator and CHatGPT doesn’t measure up in some aspects.

The good things about reliable tools is you can offload the cognitive burden onto them and know they won’t screw you over.

Almost every single post here about using ChatGPT mentions checking through its output. People don’t check though the output of their calculators.

Surely this is "just" a matter of teaching the LLM to recognize this is a job for say Wolfram Aplha and generate a query to it, then feed the response back to you?

I suspect that humans have an accuracy lower than 99.999%, and are similarly capable of producing confidently incorrect results.

GPT has a lot of hype and hysteria around it, but demanding 100% accuracy from it is a bit over the top imo. It doesn't need to have 100% accuracy on any arbitrary prompt in order to be a useful and valuable tool.

"It doesn't need to have 100% accuracy on any arbitrary prompt in order to be a useful and valuable tool."

Yet anything less than represents a danger. Never underestimate stupid.

Especially don't underestimate automated stupid

Yes. WE have a higher standard for computers. But humans produce a ton of bull shit answers. Are we trying to produce an 'answer machine' or a 'human mimic'. Because the bull shit makes it more human. And a wrong answer does not mean it's broken, what goes on in its neural net to produce a confident wrong answer might be similar to our own.

have you ever got on an airplane? are you aware that they are not 100% safe?

airplanes do not execute in a vacuum eg: computer. There are external factors that might cause an airplane to fail. A fully working airplane all things equal in terms of weather and a good pilot, will be safe.

you're missing the point. my point was: if everyone waited til something is 100% safe/complete/satisfaction guaranteed, there would be no progress in human civilization. we need to be willing to take a chance on a promising technology in order to allow for further progress. imagine if everyone said i'll wait for planes to be 100% safety guaranteed before i'll get on one - we wouldn't be flying today.

Do you have colleagues, bosses or reports that are correct 100% or the time? 99.999%? I would love colleagues that are 99% accurate, I certainly am not unfortunately.

> Do you have colleagues, bosses or reports that are correct 100% or the time? 99.999%? I would love colleagues that are 99% accurate, I certainly am not unfortunately.

I see this analogy all the time in these comment sections but it's not a very good one. A person is not a tool. One of the great achievements of humanity, and in computing in particular, is that we make tools that are more accurate than we are.

I expect a hammer to deliver a hard forward blow 100% of the time. If one out of one hundred times it delivers a hard backward blow, I cannot use it on a job site due to risk of injury to the user. The same is true of a calculator being used for financial transactions. And the same is true of a LLM that would be used for drug interactions as discussed in this thread. We already have 100% accurate ways of pulling data from a drug database—it's called SQL. A tool that is not as accurate is in at least some ways a step backward, even if it's easier to use due to its natural language interface.

Except hammers already do not work as expected 100% of the time, as evident by them painfully hitting the hands of the workers in mishaps. Yet we still use them.

How is a hammer "not working as expected" if you hit your hand with it? This makes no sense. If I hit my hand with a hammer, what am I supposed to expect except causing pain and injury?

If I try to light a cigarette and accidentally set my beard on fire due to my own clumsyness, did the lighter malfunction? No, the lighter did exactly what it was supposed to do, it was my hand that didn't do what I expected.

If you want to continue with semantics, the worker did not expect to hit his thumb until the hammer bounced off of it, hence the tool did not work as expected. Until regular hammers are able to put nails into wood by themselves, we are talking about the success of the complete system that includes both the hammer and its user.

Either way, when working with humans we already deal with plenty of misses and mistakes. Programmers create 10 to 20 bugs per 1000 lines of code, 9 out of 10 businesses fail, accountants make detrimental blunders, etc.

The point is that in the end the ML systems need only to replace these already non-perfect systems. I'll refrain from judging the consequences of this as I think it's out of scope.

Nice packpedal, except now what you said makes no sense anyway because the discussion was about tools doing what you expect them to do, not "tool + human systems" doing what you expect them to. These are two different things.

Do you think a carpenter who hits themselves with a hammer blames the hammer or themselves? Or are you going to unironically tell me they would blame the hammer + human system?

Do you have a more specific point you're trying to make?

The discussion is about AI partially replacing human colleagues. In practice, people already are not 100% reliable. You make a reasonable request and someone makes a stupid blunder instead. That's the "hammer" hitting your thumb. Maybe you were not specific enough or maybe they didn't listen but the damage is done.

Our work processes already take mistakes and iterative refinement into account. If AI, in some specific niche, is cheaper and makes no more mistakes than humans do, it gets the job.

It doesn't need to be perfect or perfectly reliable. Some guardrails will be built into it, and we'll come to trust over time.

>not "tool + human systems" doing what you expect them to. These are two different things.

Please explain how?

>Or are you going to unironically tell me they would blame the hammer + human system?

As your tooling gets more complex, yes it is very easy to have a non-zero blame assignment to each party. Look at any human+machine system where complex failure conditions can occur.

You want me to explain the difference between a hammer and a human wielding one?

One is a hammer. The other is a human wielding a hammer.

> Except hammers already do not work as expected 100% of the time, as evident by them painfully hitting the hands of the workers in mishaps. Yet we still use them.

The hammer did work 100% as expected. It's the human, who is fallible, that hit their hand with the hammer. My analogy stands. We make mistakes, we want tools that do not. LLMs should not be compared to humans, they should be compared to other tools.

To a company a person is a tool. "Human resources" means exactly what it says.

Gpt is not a calculator. I suspect that it could solve calculator problems if it was given a calculator though.

So connect GPT4 with calculator using LangChain, problem solved!

I'd be very careful assuming this, if you read the manual from OpenAI themselves they clearly provide warnings on this. It has become a vary more convincing liar apparently.

That's not me saying it, read the "System Card".

Thats why im not falling for the hype again, i think i have seen like 2 previous AI hype cycles and all those fell off after like 3 months. Same with full automatic driving stuff.

This is a black swan event which, if anything, vindicates the hype from dreamers who were ahead of their time. They had a lot of the theory right but lacked the horsepower. This will complete the last mile for other AI tech and be the lacquer that gives it the all important finishing touch.

"This is a black swan event"

By definition, black swan events are unpredictable. This example isn't that.

LLM emergence was unpredictable in 2017. Even for the best AI people.

ChatGPT is nothing more than long-form autocomplete. The specifics of approach are irrelevant.

A human brain is nothing more than long-form autocomplete. The specifics of approach are irrelevant.

Exactly. All the people saying LLM's aren't important because it is just auto-complete, really don't connect the dots that humans are also just auto-complete.

What on earth are you talking about?

Our brain excels at pattern matching. We attribute a lot of human abilities to “intelligence”, but the line between that and pattern-matching (or “autocomplete” ad it’s being referred as here) is being challenged by these latest LLMs.

If it can look at a picture, explain what’s in it, and hypothesize about physics inside the picture’s environment, is it “just pattern matching”?

"If it can look at a picture, explain what’s in it, and hypothesize about physics inside the picture’s environment, is it “just pattern matching”?"

That's extrapolation via approximation. Computers synthesize a specification. The difference is nuanced and entirely contextual.

Those are way too broad terms for this conversation to go anywhere. The point is, what's to say the LLM is not doing "extrapolation via approximation" or some process that is analog to how we think? We barely understand how the brain works to start with.

But the human brain is a much more mature product (although I will concede there are still a ton of bugs to work out that we really have no idea how to repair because the product is extremely complex and still pretty fragile).

Everyone likes to say this, but no one seems to provide evidence. I don't believe it.

Why do you think that?

I knew something changed after AlphaGo. Compute could do what we thought only true intelligence can do. So I agree, LLMs are not a black swan. It will change everything, nevertheless.

I keep reading these opinions that LLMs are just doing some advanced form of copy paste. Actually, we don't know what they are doing. Are they actually doing some form of modelling and abstraction? Seems likely to me.

"I keep reading these opinions that LLMs are just doing some advanced form of copy paste. Actually, we don't know what they are doing. Are they actually doing some form of modelling and abstraction? Seems likely to me."

This is exactly the problem with AI. For business or government, the answer is as important as the methodology employed. A black box does not work for the majority of use cases.

Until it can show its work, it's a sideshow.

Humans are also "black boxes" we don't fully understand. It's worked pretty well.

This is a shallow counterargument that entirely misses the point.

This is not even an argument, only a disagreement.

"This is not even an argument, only a disagreement."

It's a straw man.

The need for transparency in process is known, documented, and undisputed. Your comment has no relevance. My brain might be a black box, but I can still communicate and/or document the specifics of a process.

Can [insert your preferred model] do that? Didn't think so.

Given the changes in health over the past century and a half as we've adopted "science" into food, I'm not sure that I agree.

Yes, in many situations it is important that we get AIs to explain how they came up with a result, and that we are confident in the correctness of the explanation. That is probably the most interesting thing you can work on now. Is that a sideshow? I don't think so, although I wouldn't mind that at all.

AlphaGo is basically the same as image recognition, not sure why you thought AlphaGo was special compared to detecting faces. AlphaGo works the same way but maps game structures on a Go board instead of mapping facial structures and comparing those to known people.

It was able to beat the top GO player. What does it matter if that technology is also used in image recognition. I'd say that if its 'brain' can play games and recognize people, or in case of ChatGPT4, can talk and play chess, this is all starting to sound pretty 'General' to me.

That's exactly the kind of attitude I am talking about. You can rationalise pretty much everything and keep moving the goal post. I've seen AI do something nobody thought was possible. If that doesn't influence how you think about AI I cannot help you.

Nobody?

Well, maybe the self-driving car people thought it was possible. But I don't take them seriously.

next bit prediction, or long-form autocomplete, may lead to general intelligence. http://mattmahoney.net/dc/rationale.html

The black swan event includes the unexpected emergent capabilities of LLMs, which can pass professional examinations and exhibit many attributes of general intelligence, combined with the broad availability to market and massive pressure on big industry players to adapt and innovate or die.

If things keep going this way then pretty soon the black swans are going to outnumber the white ones.

I don’t put much stock in the claims about GPT4 “passing” professional exams. Many copies of previously administered exams are available, and the exams are formulaic in their construction (to make them stable, predictable targets).

A compressive copy of the internet brute-forcing its way through an exam (which it may even have digested already) is really not interpretable as performing well on the exam. It’s a meaningless measure because the tests were not designed with this use in mind.

Single stories of LLM success are just as problematic as single stories of LLM failures. As always. The fact is that we have a dangerous tool at hand that absolutely REQUIRES skepticism if you are trying to get *facts* out of it.

I agree that LLMs are extremely likely to impact many areas of work, particularly bullshit work. But as it stands you absolutely cannot use them as fact machines, the results can be catastrophic.

What it does well, among others:

- Scaffolding text, breaking writers block etc.

- Compose basic texts from minimal input, for example for bullshit tasks -> I generated an internal "vision statement" during a Miro workshop for my team by inputting a bunch of bullet points gathered from the team members brain storming. It created a concise, fluid text that everybody liked. It's now the vision statement.

- Point you in good directions, give you ideas

What it does NOT well, among others:

- provide factual responses. all responses MUST be scrutinized because they are likely containing false information. This is very dangerous for society ("Can I take this medicine with this other medicine?")

- Compose creative texts that are coherent and novel. ChatGPT texts can be quite fun but they rarely make sense beyond very superficial screening and convey no deeper message.

However, ChatGPT-like tools are used with a lot of naivety and often blind acceptance instead of using them as tools to aid your work.

For the most part I agree, but will qualify that the caveats you listed apply to the out of the box version of ChatGPT. I expect these limitations will be overcome by using it as programming substrate and connecting it to other models and APIs.

I am impressed by what we’ve seen from ChatGPT so far, but am especially excited to see what industry does with LLMs as new type of building block.

Fully agreed! I suspect that there will always be this subtle risk of catastrophic failure though, something observed in a lot of AI systems. The scrutiny filter may become less relevant but will likely not be less needed to prevent 1 in 10,000 bad responses.

If 100,000 people ask critical questions then 10 people might run into potentially catastrophic consequences. ChatGPT is a powerful tool and will only become more so but it will probably not be perfectly reliable by any means due to the nature of the system.

I am excited for the generative AI future and whatever the hell is still coming. Only those who adapt will survive.

Not sure if you tried GPT-4 but my experience with 4 is quite different. It has been quite bullshit free, though not completely. For example I asked it to contrast oral and injectable semaglutide formulations. It did a bang up job. One thing I always do is ask it for evidence. And then I look the references up. Sometimes the references don’t say exactly what it said they will. I come back and have a discussion with it. It’s a back and forth for sure, but it’s an order of magnitude more informative and at least as accurate as a google search result is for me. I come out of it learning much more than I wanted to every single time. It’s pretty much part of my daily work and life, and I spend hours on it hitting the api limits now. So I’m not sure what others are telling about how it’s still shit.

If this AI system progresses no further, it’s already transformative. It’s an intelligence multiplier for some (I’m squarely in that category, just not sure if it’s 2x or 5x), but clearly for a lot of others it’s going to be something that takes away their livelihood.

I already use it for my coding work, one-way only, since I can’t paste proprietary code into it yet. The day something gpt-4-smart can plug into my orgs codebase, each of us will get at least 2x more efficient, conservatively.

* * *

> The day something gpt 4 smart can plug into my orgs codebase, each of us will get at least 2x more efficient, conservatively.

Can't wait for GPT-4 VSCode integration. What I want most is to have it see the errors and files (file formats, directories, etc) so it will automatically know what is where and how it is structured. Not just code, but also data files.

In the mean while I am starting to format my code in such a way as to contain this information, put there by me by hand. Fully documented files are better for GPT.

Just ask GPT-4 to write a vscode plugin for itself.

There’s no way out of sending the code to gpt-4 which as of now is a no-no.

> One thing I always do is ask it for evidence. And then I look the references up. Sometimes the references don’t say exactly what it said they will.

Can you give an example? I'm surprised there's room in there for actual resolvable URLs or paper references.

If you’re discussing a scientific topic just ask it for references, it generally gives the first author, year and journal along with the title, for most reasonably cited papers!

but it's very good for experts or those who're very knowledgeable and want to dig quick information / bootstrapping something, since they can digest whether the information is correct, useful, or wasteful.

You should never medicate yourself after using a LLM, not in 2023 anyway! You should not run any code you have not vetted, or take any pills you don't know what they do.

Thats the problem, most people do take pills without knowing how they work, because doctor told them to take them. Also no one (probably) is vetting all the code they run on their computers. So this advice is kinda victim blaming. You ran the code that you did not verify? too bad, your own fault. You took advice from LLM? too bad, your own fault. You got tricked in the street by a crook? Too bad, your own fault.

So what are we discussing, is, should we try to put guardrails around dangerous things so that inexperienced/vulnerable people would not get damaged.

No, I verify the code I generate with chatGPT or Codex. I don't verify code I download.

I mean all the code that you run on your computer. Not just generated by AI. I think parent meant the same.

You didn't use version 4 did you?

Because if feel this is pretty much already out of date. 4 does this _a lot_ less!

And 5, 6 or whatever will probably be better, so i don't even really get the point here.

I find there are two kinds of people in the world. First are saying that LLMs are bullshitting. The second are bullshitting about whatever LLMs are saying.

I find there are more than two kinds of people. Among them some are skeptical and some are not. The latter are good at exploring, even if blind alleys while the former keep them in check. We’re lucky we have some variety otherwise we’d risk go full force into dead ends or we’d ignore fruitful possibilities.

The duality of mankind. On one hand, we have skeptics asserting that LLMs are masters of malarkey. On the other, we've got those regurgitating the verbal gymnastics of said LLMs. But lo and behold, there exists a third breed: those who delight in deciphering the eloquent dance between the bullshitters and the bullshitted.

I asked it "What is the accuracy and confidence level (you must rate it out of 10) that the above answer is correct ?"

It always generates a random number when it is caught making up bullshit sometimes claims that the information 10/10 accurate and LLM's always provides accurate information.

OpenAI did exact the test for GPT-4. The raw, non-fine-tune GPT-4 is quite good at predicting confidence level ("highly calibrated" by their words). But the RLHF fine-tuning process seems ruin its calibration. Figure 8 on page 12 of GPT-4 Technical Report shows this dramatic changes before & after fine-tuning.

I've been using Bing AI instead of Google for a few days to test it out. It gives responses with citations. A few times already it has hallucinated details that aren't on the cited website at all, but that do sound very plausible (for example something about the horns of a bull pointing up when I asked about the etymology of "bullish".)

I think it comes down to the fact that most stuff is actually like 90% bullshit so when you train a model on a corpus of everything you end up with a decent bullshit generator. Which is fine for many purposes but I'm not sure it will take over search.

It is telling that ChatGPT by training on online discussions hasn’t learned to say “I don’t know”.

I guess the "I don't know"'s are usually silent.

Lmao. Now I’m imagining any questions posted anywhere like Ask Hacker News, Ask Reddit, or Quora being filled with everyone that doesn’t know the answer replying “I don’t know.”

Product q&a on Amazon has this. It is because Amazon emails you other people's questions about products you have previously purchased, and many users don't realize there is no need to reply just to be polite.

I wish that Amazon would put a note suggesting the if you don't know, don't answer the question... I know they send emails to people that bought something when questions come up... but for the sake of sanity, why do people actually input an "I don't know" answer?

I've said this already today on another comment, but the manual from OpenAI themselves pretty much says the thing is not to be trusted for pretty much anything.

Same for Bing chat.

I've had it struggle to give me citations at all sometimes. Once I even had it give me a citation to a C# webpage for an Elixir question

i agree citations dont always support bings conclusion, thats been my exp as well

ChatGPT is like the robot from Asimov's short story Liar!: It doesn't tell you the truth, it tells you what it thinks you want to hear. Instead of reading minds it returns a statistically plausible response to queries similar to yours.

Stop evaluating tech by misapplying it. ChatGPT is not really good or reliable at factual questions, yes. But it's fantastic at transforming input data that you provide to it.

This has been my experience: Chat GPT is astonishing in terms of producing plausible natural language text that looks related to the question. That's an astounding achievement imho. It does not (and I suspect wasn't expected to) produce correct answers to questions.

I had an extended back-and-forth in which I asked ChatGPT how to undo mistakenly marking a message as junk in Messages on iOS.

It gave me a 7-step answer that was completely wrong. Then I said no, you can't do that, it apologized and gave me an 8-step answer that was completely wrong. When I pointed out in this case there is no "Junk" folder in iOS Messages, it apologized for the confusion and gave me another 8-step answer that was completely wrong. When I pointed out why that one wouldn't work, it gave up and said recovery was impossible and that I would need to contact the sender to re-send, and be more careful when marking messages as junk. This was still wrong, as recovery is possible, just not by any of the means it described.

So yeah, I have been super-impressed by the quality of output from these LLMs, but I cannot imagine actually relying on one for anything where correctness matters.

Giving me a nice list of Korean shoegaze bands, sure. Its step-by-step for how to become a better volleyball player will be great for my daughter, and was better than the answer to the same question from Google. But correctness? No.

I think AI's hype machine is definitely going to negatively affect its future. If people are made to believe it's able to automate everything and it gets just one crucial thing wrong, the loss of credibility at a time when it's supposedly a finished product will be considerable. If it gets something wrong and it's made clear it's still under development the reaction will be less negative. Currently the hype puts the current state of AI somewhere between Rosey the robot and Skynet. I don't think the realization of how far we are from either will do much to promote adoption.

> It made up a severe risk of death taking a very common medicine combo.

Probably trained on too many of those SEO hyper optimized medical sites people get when they search symptoms and such.

It's a great bullshit generator.

I think upper management needs to be more scared of the implications than ICs