> I would set traps to ensure that they are doing that. ;)

You won‘t have to. Students are extraordinarily skilled at fining endless pitfalls on their own.

That can be a double edged sword, particularly when the lesson is about an edge case/pitfall. You'd be surprised how they can seemingly blunder their way to the solution while waltzing around the edge case, and thus not learning the intended lesson.

Setting the stage where they either land in the pit, or start from within, ensures they're interacting with the material as intended.

> My experience with LLMs so far is that they very confidently give you an answer and very convincingly explain that answer but for certain portion of the time that answer is very wrong.

This is true of humans just as well

Whilst that is true, I feel that if that human is an educator employed by a reputable institution, answering questions in their field, the expectation of most is going to be that they provide accurate information. As this is an introductory class, any arguments that even experienced teachers can occasionally cling to incorrect or outdated information becomes moot in my opinion.

LLMs as of now simply appear not reliable enough for an institution like this one to attach their name (which brings credibility in the eyes of the public), especially as their comments on the need for users to be vigilant ("they should always think critically when taking in information as input") make it sound like they did not in fact find some new way of reliably grounding output in accurate information.

Mind you, this is coming from someone who has great hopes for the future of LLMs in education as they can more effectively deal with students on an individual level. I just feel that currently LLMs are still too error-prone for an institution such as Harvard to incorporate them into an introductory course to assist. Until these insitution stop putting the majority of responsibility on students finding erroneous output, I feel this is not ready for wider educational use.

> an educator employed by a reputable institution,

What's a reputable institution? These days most "experts from institutions" are proven to be wrong. The COVID-19 pandemic was a clear eye-opener on how little you could just trust anyone.

Humans are not programmed to avoid saying "I don't know."

Oh really? You will find very few people admit that they don't know something.

If you ask someone they will give you their approximation of an answer. However if you told them if they would bet money on it, and promptly make them do so, you will find them saying IDK more frequently.

Even this, I wouldn't bet money on what I just said.

There is a prominent culture where this is such a common occurrence that you have to be careful about accepting their confident assertions.

Humans also tend not to write "I don't know" into text content about a certain subject. Usually the idk is hidden by not writing at all

One useful trick to avoid some of the pitfalls of ChatGPT for non-creative tasks is to respond to its output with something like:

_"Point out incorrect assumptions or statements in the above answer."_

I got the idea from Khan Academy's [implementation](https://youtu.be/A7REVn9gzgs?t=1208) of GPT-4, where they allow it to 'think' by generating an internal response first, before generating a final response. This apparently improved the veracity of its output by a significant margin.

If ChatGPT doesn't double down on its original answer, then I know definitely not to trust it.

This doesn't solve the underlying problem, because it doesn't ultimately correct the error in a reliable way, but it does help me avoid some of the more obviously garbage output.