For a solution "now" to the cheating problem, regular exam conditions (on-site or remote proctoring) should still work more or less the same as they always have. I'd claim that the methods affected by LLMs are those that could already be circumvented by those with money or a smart relative to do the work for them.
Longer-term, I think higher-level courses/exams may benefit from focusing on what humans can do when permitted to use AI tools.
Two decades ago, when I was in engineering school, grades were 90% based on in-person, proctored, handwritten exams. So assignments had enough weight to be worth completing, but little enough that if someone cheated, it didn't really matter as the exam was the deciding factor.
> Return to the old "viva voce" exam? Still used for PhDs. But that doesn't scale at all.
What? Sure it does. Every extra full-time student at Central Methodist University (from the article) means an extra $27,480 per year in tuition.
It's absolutely, entirely scalable to provide a student taking ten courses with a 15-minute conversation with a professor per class when that student is paying twenty-seven thousand dollars.
I'm currently a PhD candidate, and our program includes separate written and oral qualifying exams during the first year or two, and a public defense of the dissertation at the end. I thought some minor variation of this was nearly universal.
It's also my observation, by the way, that the public dissertation defense (and even the written dissertation itself) is less of a big deal than outsiders tend to think. What matters is doing the research that the advisor / committee wants, and working on some number of papers that get accepted into workshops / conferences / journals (depending on the field). Everything else seems to be kind of a check-the-box formality. By the time the committee agrees that someone has done enough to defend, it's pretty much a done deal.
My particular school and department is interesting because it doesn't have any hard requirement for publications, and it aims to have students finish a PhD in about three years of full-time work (assuming one enters the program with a relevant master's degree already in-hand). There has been some tension between the younger assistant professors (who are still fighting for tenure) and the older full professors (who got tenure in, say, the 1990s). In practice, the assistant professors expect to see their students publish (with the professors as co-authors, of course) and would strongly prefer to see a dissertation comprised of three papers stapled together, regardless of the what the school and department officially says. The full professors, on the other hand, seem to prefer something more like a monograph that is of "publishable" quality, maybe to be submitted somewhere after graduation. They argue that the assistant professors should be able to judge quality work for themselves instead of outsourcing it to anonymous reviewers. Clearly, there are different incentives at play.
We would need to increase the amount of teaching staff by well over 10x to do this. The costs would be astronomical.
> The costs would be astronomical.
Those 200 students have paid the college $549,600 for your class.
The costs are already astronomical.
Is it so unreasonable for some of that money to be spent on providing education?
The students paid me nothing. The university provides some TAs, that's it. But even if they gave me all of that money in cash to spend, this would be totally impossible.
I'm supposed to grade a student based on 1 conversation? Do you know how grading and teaching work? Can you imagine the complains that would come out of this process? How unfair it is to say that you have one 15 minute shot at a grade?
But fine, even if we say that I can grade someone based on 1 conversation. What am I supposed to ask during this 15 minute conversation? Because if I ask the every student the same thing, they'll just share the questions and we're back to being useless.
So now I need to prep unique questions for 200 people? Reading their background materials, projects, test results, and then thinking of questions? I need to do that and review it all before every session.
Even with a team of TAs this would be impossible.
But even if I do all of this. I spend hours per student to figure out what they did and know. I ask unique questions for 15 minutes so that we can talk without information leakage mattering. You know what the outcome will be? Everyone will complain that my questions to them were harder than those that I asked others. And we'll be in office hours with 200 people for weeks on end sorting this out and dealing with all the paperwork for the complaints.
This is just the beginning of the disaster that this idea would be.
It's easy to sit in the peanut gallery and say "Oh, wow, why didn't my arm surgery take 10 minutes, they just screwed two bones together right?" until you actually need to do the thing and you notice that it's far more complex than you thought.
> The students paid me nothing.
Well gee, there I was thinking they were paying $27,480 per year for "tuition"
Later, people learned that and started to buy diploma: faster, cheaper, no risk of failing the final exam.
We all know they'll just exploit grad students rather than hire real teachers.
My office mate in engineering was trying to get funding for a start-up. I was trying to get a consulting business off the ground. Neither of us achieved those things, but whatever. He got a teaching gig at the community college, which is unionized and actually a pretty good situation. I found a regular day job through his network.
A friend of mine had an adjunct gig in the humanities, and used his off-time to learn how to code.
A lot of academic spouses get adjunct gigs, especially if they want to balance part time work with child care.
A lot of adjuncts sit around in precarious financial situations, developing serious mental health issues, and drinking problems because the system taught them that this is a form of success.
Going to industry and making money? That's failure. That's an "alternate career". Not scraping by in a system that couldn't care less about you. That's success.
It's pretty vile. I've never had a student become an adjunct. It would be a personal failure that I haven't given them the tools to thrive.
Where do you get this from? The people I know with pure research positions get paid basically the same (after correcting for 'rank' and seniority) as those who split their time between research and teaching.
The top earning professors in the nation in mathematics are all very good research mathematicians
That threat may be enough to dissuade students from cheating with AI.
Countries have no problem spending astronomical amounts on old people. If the country wants productive young people, the country will find a way.
Why bother training and educating the young people who are already here when you can just import them from poorer countries?
But in principle this is not a problem for me, I already spend at least this much time grading papers, and an oral exam would be much more pleasant. The real problems will come up when (1) students are forced to schedule these 15-minute slots, and (2) they complain about the lack of time and non-objective grading rubric.
I completely agree, but then again it seems to me that society also functions according to many norms that were established due to historical context; and could / should be challenged and replaced.
Our education system was based on needs of the industrial revolution. Ditto, the structure of our working week.
My bet: We will see our working / waking lives shift before our eyes, in a manner that's comparable to watching an avalanche in the far distance. And (similarly to the avalanche metaphor) we'll likely have little ability to effect any change.
Fundamental questions like 'why do we work', 'what do we need' and 'what do we want' will be necessarily brought to the fore.
I think that we'll see fundamental changes, but it will be based on cheaper consumer goods because all of the back end white collar labor that adds costs to them will be (for all intents and purposes) free.
But we will see the absolute destruction of the middle class. This will be the death blow. The work week will change, but only because even more people will work multiple part time jobs. We'll think about what we need, but only because we'll have cheap consumer goods, but no ability to prepare for the future.
I think it's bleak. Source: most of human history. We're not, as a species, naturally altruistic. We're competitive and selfish.
Looking back on it, I think it could be weirdly precient.
Two classes of society; one living a life of leisure, the other fighting on the plains.
(.. maybe minus Connery in a mankini)
The Time Machine is a very famous example.
All the low paid, physically laborious work is not affected by AI, so there will be plenty of work, especially with aging populations around the world.
The question is will it be worth doing (can the recipients of the work pay enough) without being able to provide the dream of being able to obtain a desk job for one’s self or their children.
Low paid physically labourorious work has been increasingly automated since the wheel started helping potters.
The history of computers often ends up referencing Jacquard looms because it used punched cards.
LLMs have been demonstrated controlling various robots.
AI's coming for all jobs, it's just a question of which jobs cease first.
There is a sad, depressing world out there. One of my parents work at an elderly home, and the shit that happens there is just wild. Zero responsibility and accountability. Deliberate killing of people out of pure inconvenience, etc.
I am in favor of a "social support network".
[1] Requires money, e.g. pension, which is increasingly less, and they keep increasing the age.
I wish euthanasia was legal here.
When I was in university (Humanities degree), we had to do lots of mandatory essays throughout the year but they counted little towards your overall mark, maybe 10% iirc.
The majority of marks came from mid-year & end-of-year exams.
A simple change to negate AI is to not award any points for work outside exams — make it an optional chance to get feedback from lecturers. If students want to turn in work by AI, it's up to them
My sense is that if writing was entirely optional, it would be too easy for students to let it slide; having 10% count toward your grade is a good "nudge" to get honest students to actually do the work. I'd just give a little admonition at the beginning saying that I'm not going to bother checking if you use LLMs, but if you do you're an idiot, because you'll be completely unprepared for the hand-written essay you have to compose yourself in the exams.
University students should be capable enough for that, different at secondary level
It doesn't scale if performed by a human. But what if... we employ AI to conduct the voice exams?
My current best guess, is to hand the student stuff that was written by an LLM, and challenge them to find and correct its mistakes.
That's going to be what they do in their careers, unless the LLMs get so good they don't need to, in which case https://xkcd.com/810/ applies.
> Personally I think all this is unpredictable and destabilizing. If the AI advocates are right, which I don't think they are, they're going to eradicate most of the white collar jobs and academic specialties for which those people are being trained and evaluated.
Yup.
I hope the e/acc types are wrong, we're not ready.
Finding errors in a text is a useful exercise, but clearly a huge step down in terms of cognitive challenge from producing a high quality text from scratch. This isn't so much an alternative as it is just giving up on giving students intellectually challenging work.
> That's going to be what they do in their careers
I think this objection is not relevant. Calculators made pen-and-paper arithmetic on large numbers obsolete, but it turns out that the skills you build as a child doing pen-and-paper arithmetic are useful once you move on to more complex mathematics (that is, you learn the skill of executing a procedure on abstract symbols). Pen-and-paper arithmetic may be obsolete as a tool, but learning it is still useful. It's not easy to identify which "useless" skills are still useful as to learn as cognitive training, but I feel pretty confident that writing is one of them.
I disagree.
I've been writing a novel now for… far too long, now. Trouble is, whenever I read it back, I don't like what I've done.
I could totally just ask an LLM to write one for me, but the hard part is figuring out what parts of those 109,000 words of mine sucked, much more so than writing them.
(I can also ask an LLM to copyedit for me, but that only goes so far before it gets confused and starts trying to tell me about something wildly different).
> It's not easy to identify which "useless" skills are still useful as to learn as cognitive training
Indeed. And you may also be correct that writing is one such skill even if only just to get the most out of an LLM.
What I'm describing here is very much a best guess from minimal evidence and the current situation; I would easily drop it for another idea if I saw even very minimal evidence for a better solution.
Please expand?
Extremely optimistic about the benefits of new tech, downplay all the risks, my experience of self-identifying e/acc people has generally been that they assume AI alignment will happen by default or be solved in the marketplace… and specifically where I hope they're wrong, is that many seem to think this is all imminent, as in 3-5 years.
If they're right about everything else then we're all going to have a great time regardless of when it comes, but I don't see human nature being compatible with even just an LLM that can do a genuinely novel PhD's worth of research rather than "merely" explain it or assist with it (impressive though even those much easier targets are).
<< But that doesn't scale at all.
I realize that the level of effort for oral exam is greater for both parties involved. However, the fact it does not scale is largely irrelevant in my view. Either it evaluates something well or it does not.
And, since use of AI makes written exams almost impossible, this genuinely seems to be the only real test left.
Isn't it easy to prevent students from using an AI if they are doing the exams in a big room? I mean when I was a student, most of my exams were written with just access to notes but no computers. Not that much resources needed to control that...
That is not say there won't be cheaters ( they always are ), but that is what proctor is for. And no, I absolutely hated the online proctor version. I swore I will never touch that thing again. And this may be the answer, people need to exercise their free will a little more forcefully.
Happens in tax filing too.
On the contrary; with AI it scales better than ever before.
(AI alone for assessment is even worse than AI proctoring!)
This will be it. [edit: for all education I mean, not just college] Computers are going to become a bigger part of education for the masses, for cost reasons, and elite education will continue to be performed pretty much entirely by humans.
We better hope computer learning systems get a lot better than they’ve been so far, because that’s the future for the masses in the expensive-labor developed world. Certainly in the US, anyway. Otherwise the gap in education quality between the haves and have nots is about to get even worse.
Public schools are already well on the way down that path, over the last few years, spurred by Covid and an increasingly-bad teacher shortage.
That basically never happens and the outcome is the result of some sort of struggle. Usually just a peaceful one in the courts and legislatures and markets, but a struggle nonetheless.
> new methods should be used to assess student performance.
Such as? We need an answer now because students are being assessed now.
Return to the old "viva voce" exam? Still used for PhDs. But that doesn't scale at all. Perhaps we're going to have to accept that and aggressively ration higher education by the limited amount of time available for human-to-human evaluations.
Personally I think all this is unpredictable and destabilizing. If the AI advocates are right, which I don't think they are, they're going to eradicate most of the white collar jobs and academic specialties for which those people are being trained and evaluated.