I agree that professors can probably develop ai resistant evaluations.

But there is minimal institutional support for this. Everybody is going it on their own. And the ai tools are changing rapidly. Telling a ntt faculty member teaching a 4/4 that they just need to stop being lazy and redo their entire evaluation method is not really workable.

Iteration is also slow. You do a new syllabus to try to discourage cheating. You run the course for a semester. You see the results with now just a few weeks to turn around for your spring semester courses. At best you get one iteration cycle per six months. More likely it is one per year since it is basically impossible to meaningfully digest the outcome of what you tried and to meaningfully try something different mere weeks later.

I've been urging this kind of adaptation for the past couple years, with very little success. Switching to evaluation by only on-site work was simple to do, but creating new work forms has been largely a non-starter. Young professors are focused on their publication count and most old codgers (like me) have their eyes on retirement and hoping to escape before accountability hits the fan. When I offer to help explore restructuring their teaching materials, the typical response is glazed eyes and sudden awareness of the need to be elsewhere.

Yeah, I've been thinking about when the calculator came to bear. I was in high school physics when (rich) kids started bringing in "scientific" calculators. There was a flap then about whether they should be allowed at all.

I'm not sure they are analogous to LLMs, but a compelling argument at the time was that the students were going to use them in the field anyway. (That certainly seems to be the future for software engineering students.)

equating chatgpt to "a better spell checker" is wild; how is writing not also a skill that should be taught?

It’s the thinking that precedes writing that should be the bulk of your grade. Possessing what looks like an artifact of organized thought isn’t enough these days to evaluate a student.

The “thinking that precedes the writing” is not where the gold is. That kind of thinking is often confused, internally inconsistent, liable to miss critical details or nuance, and full of deductive leaps which may or may not pan out. Writing demands rigor of thought, it forces us to question premises, find evidence for a point of view, discern between the hypothetical and the factual, and try to organize these such that they cohere.

The thinking _is_ the writing. To be able to write is to be able to think, and if you are surrendering the writing to a machine that’s not ultimately what you’re surrendering—you’re giving up independent thought itself.

You misunderstand. I mean the thinking that you say “_is_ the writing”. But the final words aren’t the thoughts that you should be graded on (aka “this is not a pipe”, but your essay is not your thoughts). What a new assessment could look like is an open book debate with a human (or, hell, with a rubber duck LLM). If at the end your thoughts are organized and self consistent you’ve done well.

I think if you removed the economic incentive to cheat then humanities wouldn't have the problem it does with LLMs

There's always an incentive to cheat: economic, reputational… people cheat in single-player video games. People cheat in sudoku puzzles. By removing all incentives, all you do is destroy the concept of "cheating", not prevent the behaviour.

I think you've listed several disparate incentives that should be looked at separately.

Economic incentives are what this thread is about. If those are removed, some instances of cheating will subside.

Reputational incentives are similar and harder to address, but they are also less strong because the cheating has a much higher chance to backfire. If you are found out, your economic benefits might continue, but your reputation is immediately damaged.

Singleplayer games are an entirely different situation. There's clearly no economic incentive here, and reputational incentive only applies if you are looking to share your singleplayer result with other people, in which case I could argue it's no longer singleplayer in the game theoretic sense. If it's not that, the only remaining motivation to cheat at something like Sudoku or a singleplayer videogame is to learn or at least satisfy an itch or curiosity, which I think is a perfectly legitimate motivation.

>Economic incentives are what this thread is about.

You cannot remove economic incentives for the most part, 'economy' is attached to all facets of peoples lives, especially where rival goods exist.

You forgot laziness.

It’s definitionally not cheating if it’s a single player environment. That’s like saying skipping to the end of a movie is cheating.

What needs to happen is a return to a more Socratic method of teaching the humanities.

Instead of being about the consumption and production of texts -- texts which only the TA and maybe the prof will read ... ever -- the class should be more about the dialectical process of discussing the ideas in the texts and lectures the students consume.

An LLM will be able to easily rattle off an undergrad level paper. But it can't sit in the classroom and have an intelligent conversation with peers.

Of course none of this will happen because it doesn't scale economically nearly as well. It is far more labour intensive.