Its more troubling because its being used in a role as an educational resource. the audience is learning and significantly less able to realize that.

Firstly, the unreliability may not be permanent, and we may improve the AI accuracy in the future. Secondly, we need to figure out what works and what not, and we're in the middle of such phase. Finally, there is no need to be alarmed, even with hallucinations, the amount of advanced (and reliable) knowledge provided by the AI vastly counterbalances the small mistakes. The elitist mindset needs to die, let's give access to advanced knowledge to everybody and stop putting it behind some unreachable walls.

That is a good goal. Is AI the best tool for the job?

> Firstly, the unreliability may not be permanent, and we may improve the AI accuracy in the future.

How about we fix the unreliability first and not put the cart before the horse.

> The elitist mindset needs to die, let's give access to advanced knowledge to everybody and stop putting it behind some unreachable walls.

What are you talking about? We have high quality university courses on the internet for free. We have things like the Khan Academy and similar sites. How is advanced knowledge behind unreachable walls?

What walls are you talking about and how an AI assistant that you have to "donate" to get access to is going to help with these walls?

Chatgpt is already more accurate than a lot of my high school teachers.

My thoughts exactly. What is the baseline here? Teachers already give biased and/or inaccurate information at a rate much higher than I have seen in ChatGPT.

Heh, when Texas dictates what gets into schoolbooks.

"What happened between 1850 and 1960 regarding civil rights"

A: "Oh nothing, it was great for everyone"

History is a mess here. That said it would be nice if it were incorporated with written texts, lessons, and the AI so you can bounce questions off of it.

Is there a name for this fallacy (is it a fallacy?). I feel like we see the same issue with self driving cars crash rates. People are hesitant to adopt because they aren't perfect, but ignore that it's better than the status quo.

https://en.wikipedia.org/wiki/Nirvana_fallacy maybe?

Colloquially: "Don't let perfection stand in the way of greatness."

I'll don't think it's a fallacy. People are more predictable and can be accountable for their actions, and even admit ignorance, eg answering "I don't know ask someone else".

It's worth discussing if we want AIs replacing them - maybe the scalability is better, but these are usually worse.

> People are more predictable and can be accountable for their actions...

I would disagree. I think that at this stage, ChatGPT is very predictable, and we have excellent statistics on how likely it is to respond correctly to a particular type of question. I would have much less predictability about any particular human teacher, who might have an especially good or bad day.

Also, ChatGPT can be much more accountable, in the sense that when issues pop up, a filter can be implemented to prevent these particular issues for arising again. I don't see how humans are better in this regard - even if they are "accountable" in some vague sense, it can take many months to get rid of an underperforming teacher, and can take decades to change something across the system.

There is being misinformed and making a mistake when reasoning. Then there is making s** up.

Most humans have the decency to not do the latter while AI (at the moment) freely makes up complete lies when it doesn't have the answer instead of just admitting it doesn't know.

At least misleading instruction from teachers tends to be of a sort that others also experience, so can later be addressed en masse (e.g. mythologized history, poor-but-common explanations for things like how airplane wings work). Or spreading the same political propaganda that can be found all around, so is pretty easy to spot.

It's not usually that you ask them a question they can't answer, and they just confidently make up something entirely wrong, and possibly also an incorrect explanation that few or no other people have seen.

It's a different kind of wrong. One can write a book addressing common misconceptions picked up in school, from human teachers, and cover much of it. Does that work for lies AIs tell? They could be about almost anything, they could be broadly wrong, they could be subtly wrong, et c. "Show me some things my teachers may have been wrong about" seems to me an easier request to adequately address than "show me some things my AI tutors lied about", because the former tend to cluster around a kind of common space and form an experience with a great deal of overlap between students, while I'm not sure the latter do.

I suppose that this is true. But at least with an AI under your control (as a Khan Academy admin), you can run a query to analyze any unexpected decrease in student performance, find correlations with changes in the keywords that these students received from the AI tutor, isolate the particular misleading AI outputs by running some (possibly automated) experiments, and then once identified, fix the problem systemically by prompt engineering, fine-tuning or output filtering.

What's the analog of that with human teachers, and how long would it take to implement a solution there? Even if you had all the budget in the world, how would you go about making all teachers stop misleading students about how airplane lift works?

Problem is, we see it as authoritative source than as a regular teacher.

They're saying that they have reduced hallucinations. But RLHF seems to undo some of the improvements (figure 8 in the paper).

Those are calibration curves for confidence.

Being confident in false things is part of what makes hallucinations troublesome.

'reduced hallucinations', um ok.

There are two indicators that this is PR bullshit produced by whoever is trying to capitalize their investment, 1) is the use of the marketing term 'AI' or 'AI-powered' and 2) use of the 'think of the children' trope.

'AI' is a marketing term? I don't see how that's more than saying that 'cloud' is a marketing term - while being useful in marketing, it's just an industry term.

Don’t expect it to know specific facts but it is excellent at explaining topics

That is a feature, not a bug. It's an efficient way to brainwash a population.

Are the tolerances in education really that tight?

How many terrible teachers are allowed to continue to teach after decades of disastrous results?

How many ideologues with no interest in teaching, but every interest in indoctrinating young minds, are tolerated because the alternative is "no teacher" and the class wouldn't run?

How many teachers who would fail state exams, teach, despite relying on answer sheets to be "competent"?

I agree with you that on the top end of education, this is no replacement and at best a supplementary tool. For the poor kid in a bad neighbourhood whose teacher is more interested in "de-colonizing" mathematics than teaching mathematics, this is a Godsend.

I feel the question is, should we be making this worse with unreliable AI.

Currently AI seem to have 2 weaknesses, it’s quite bad at reasoning and it hallucinates.

Yes, humans screw up reasoning too but at least they try. AI like ChatGPT seem to skip critical thinking altogether - there are examples of ChatGPT contradicting itself multiple times in a single conversation.

Then there are hallucinations. When people don’t know something, most just say they don’t know. AI just can’t seem to help itself but make up complete BS that it confidently try to pass off as facts.

P.S. Frankly I think the 2 might be related. AI can’t “sanity check” it’s own output.

> How many terrible teachers are allowed to continue to teach after decades of disastrous results?

Counterpoint: Why "hire" a known-bad teacher? Is a teacher that is known to give fake information better or worse than none at all?

I don't know the answer to this, but I hope that Khan Academy thought about this and decided that the good of GPT is better than it's bad. Or, they can't know so thats why they're doing a limited study before rolling this out.

Right now, we hire and retain bad teachers because many public schools act more as childcare than learning centers.

I mean that is but one thing to worry about, we've gone about nowhere with the alignment problem, and we're screaming ahead at full speed with making these things more powerful.

Without providing context or examples, GPT-4 is already better at answering questions than the average teacher, with the unlimited patience that only a computer can provide.

With context, which Khan academy has in abundance due to their lesson plans and transcripts, accuracy will be higher than even the best teachers and tutors.

Once you give context and known-true facts to best-in-class LLMs like GPT-4, the output is shockingly good.

If you don't know how they are using GPT-4 it's not fair to say it will hallucinate.

As far as I understand the preferred way to use LLMs nowadays for domain specific information retrieval is through embeddings that insert the related context in the prompt. GPT-4 is specially good for this since they increased the prompt size almost by an order of magnitude.

This means that the model can be given a very specific task: to extract information from the context or avoid providing an answer at all.

The answer doesn't rely on the neural memory of the model, since it doesn't need to store information, just understand the task, and they are really good at that.