Today you learned a lesson. You will need to add to your course work how GPT can get things wildly wrong. Just a simple example should not be too terribly hard to find. Then plant the seed 'if it has that wrong what else is it getting wrong? It is a good tool of getting the general idea but one you need to audit. This class is where you are going to learn the basics and know when this thing is wrong when you use it' That thing is not going away any time soon.

But clearly it gets things right often enough to be worth using to cheat on a test or assignment.

I actually did say something more or less exactly like that pretty early into the course.

Something to the effect of "ChatGPT is pretty cool, I like it, but it will just make stuff up sometimes in extremely convincing way. It's a tool that's dangerous to newbies and powerful to professionals, and I'm hoping this class will help you get closer to the latter".

Ah good. A very small demonstration probably would drive the point home. It is one thing to hear that but another to see it yourself.

Also, how GPT can get things right, and that if students use GPT (because it's the early days of the graphing calculator all over again but now for code) then the exercise is not "write the code" but "explain the conditions under which this code doesn't work, and why" because if you're taking a programming class it doesn't matter what writes the code, what matters is whether you understand that code. So if you're turning the exercise into a code audit by using GPT, you better damn well be able to explain what problems are left in the code it generated.

> You will need to add to your course work how GPT can get things wildly wrong. Just a simple example should not be too terribly hard to find. Then plant the seed 'if it has that wrong what else is it getting wrong?

People did the same thing for Wikipedia and it was never convincing. And they eventually lost the battle. It's going to require more than that to do the same for GPT.

I wonder if it would be worth it to give some sort of in person feedback to convey (a) you noticed the dramatic improvement (b) you were even worried they might be cheating (c) you decided to give them the benefit of the doubt (d) congratulations on their hard work paying off (e) but if they did cheat they got lucky and if they keep it up it's going to backfire on them one day.

Yeah, I've drafted a few emails that I have not sent stating more or less what you listed, in "decreasingly-pissy" tones.

I think by tonight I'll have cooled off enough to write an appropriate email.

I really don't think this is a good idea.

If I got an email half accusing me of cheating I would complain to anyone who would listen.

Whilst it's unfortunate this kid might be cheating the system, I don't think it's worth being upset if they got through. The point of these exams is to allow students to study and learn these topics, which it sounds like they did. It sounds like while maybe you failed in gatekeeping the cheaters (an impossible post gpt task) you have succeeded in the real goal which was to help these kids learn and improve.

Oh I largely agree; I'm kind of learning that you have to view teaching as somewhat more of a "statistical success" than anything else.

It's just one of those things that while you're totally right, the one or two bits of failure kind of nag at me. It's extremely easy to take these things personally (especially for a nascent teacher like me) when in reality I should likely view things as transactional.

A good and reasonable sentiment. It's a very human thing for people who are largely successful at something to fixate unproductively on failures--and in this case you can't even conclusively say that it was a "failure".

I think it's good. This sort of thing will move Universities back to being about teaching and research instead of vending credentials.

I've gone back and forth about how I feel about grades as a whole.

I do feel that by having these tests and grades that it does, indeed, sort of become just a transactional way to "vend credentials" instead of focusing on learning. I also think that often homework and tests aren't great measurements; I was bad about doing homework in college the first time around, and it was frustrating to the professors because I was doing horrible grade-wise, but was extremely active during class and generally did fine on the tests. I was mostly learning the material (at least well enough to get an A on the exams), but because I wasn't submitting my assignments, they would be forced to give me a C or a D.

At the same time, I really haven't figured out a better system. I would be open alternative systems, but I think grades and grading and tests do the best job for the "average case".

Maybe I should email some of my old professors and apologize for the headaches I probably gave them...

> This sort of thing will move Universities back to being about teaching and research instead of vending credentials.

I don’t see this happening unless jobs stop caring about credentials or become more willing accept other forms of credentials.

Of course I’m assuming you aren’t specifically talking about software, which doesn’t have nearly the amount of credentialism as many other fields.

Honestly, unless the grade for that class translates into taking something directly tangible from other students, and a "java" class sounds pretty intro, I wouldn't give any thought to it. Cramming for exams and then immediately forgetting everything you hastily memorized is a-okay even though you didn't learn anything either.

A random college class, to me, isn't the kind of thing where cheating really matters. The stakes are low, it's expected that most students will do well, and everyone gets the same degree at the end. If someone wants to sabotage their own education then fine, and if it doesn't bite them later good for them.

Yeah, I'll admit that the problem is somewhat on my end, but I find it very hard not to take these kinds of things personally. I agree that I shouldn't take them personally, but that's easier said than done.

I'm starting to see this come up in coding interviews.

I had one candidate that struggled MIGHTILY on a coding problem that was easy (kept running into syntax errors) but then pulled out an esoteric solution for the second problem in no time flat.

I can't prove that he was cheating but it very highly suggestive that he is.

I hate it, but I'm starting to think I have to do an AI detection question now. It's not hard, just ask someone to do something impossible. However, I don't like the fact that now I need to be "tricky". I've never believed in making coding challenges hard, I just want to see if you can write code.

I saw something similar in a recent interview. I brought in a candidate who did not meet all of the typical resume check boxes, but who had a long history of success on highly technical problems. These type of candidates can bring very new perspectives to our team, but they’re often a long shot in interviews.

In this case the candidate chose to use a language that I don’t know well and they said they weren’t especially familiar with either. Then they struggled a lot to even talk through the question and it was a grind to work through each line of code. I couldn’t offer much help, and they were very clearly unsure at each step.

But when we clicked “run” on the code, we were both surprised to see that it ran on the first try. Not only that, but it already worked for all of the “but what about…” follow up edge cases. And the candidate didn’t know why… “oh, I guess the language just works that way.”

I don’t know if they were cheating, but the entire thing was far outside any other interview experience I’ve ever had — and fits the pattern of what I’d expect an LLM to produce.

this is me using ocaml

Seems like there's a silver lining here. It reinforces the idea that the answer to the problem is not necessarily the important part. When you can trivially provide a correct answer, it places more emphasis on the skill of being able to actually explain and work through the logic as a way to distinguish yourself from other candidates.

You didn't hire them, right?

I'm a pretty good practical programmer, good at noticing potential edge cases, writing code that's changeable without being overcomplicated. This based on feedback across years of professional work.

I also have adhd and literally brain damage and have worked professionally in about a dozen different programming languages. I can't write a for loop or declare a static method or w/e in any of them without googling the syntax or using my editor's hints & autocomplete.

That's fine and all, but not what was happening.

They had google access and editor hints. Further, I start every interview with "please, ask any questions, we aren't trying to trick you we just want to see if you can code." And I mean it. If someone has a minor problem with remembering syntax or how to declare it I don't really care and will happily let them know how to do that.

Heck, I've even had interactions where a candidate was like "I think there's a method that does x for this" in a language I was unfamiliar with, so I googled up what x was and shared it with them mid interview. "Oh yeah, looks like this is what does x for your language".

There was also some pretty weird behaviors with the camera/screen that caused them to need to touch it fairly frequently (I'm guessing to take over keyboard control from the AI software they were using).

What they were doing, repeatedly, though the interview was writing

`a[i]` then having to change it to `a.get(i)` because they were working with a `List` in java. I get maybe doing that once or twice, but in the course of the interview they did it every single time they needed to pull something out of the list. (and each time needed to touch their screen and take a few minutes to correct it). This is why I strongly suspected cheating. It extended a question that normally takes 15 minutes to 35 minutes. Then the question I ask that usually takes ~45 minutes they completed in 15 with an optimal solution that relies on a data structure I literally only learned about because the interview hints for the question are like "Hey, java has this data structure that makes everything easier".

Yeah I get it. This is one of those things that's really hard to convincingly describe online because each individual thing is explainable eg I will write a[i] every time and my format-on-save will change it to List.get(a, i) every newline or whatever.

But you were there and I wasn't and if the overall situation seemed suspicious to you that's evidence at least as strong as any specific thing. Intuition can certainly pick up on weird behavior even if it's hard to explain how after the fact.

This may be an unpopular point of view, but I think in situations like this, you should just pass on them. They are applying to tens of jobs and your company is interviewing tens of candidates, so there's no real reason for either of you to continue this process if they struggled on the easy problem and had questionable reasoning on the hard problem. However, if the candidate could very clearly explain the esoteric solution, there is a good chance that it's just similar to something she has done before and you should not be suspicious.

I am assuming that you had a compiler or IDE in this situation providing a useful error message. If you were whiteboard coding and you were telling the candidate "you have a syntax error but I won't tell you where" (like several folks at a former employer of mine were fond of), you may be the problem, not the candidate.

We ultimately did pass.

> I am assuming that you had a compiler or IDE in this situation providing a useful error message.

Yup. I prefer to give candidates compilers/IDEs and even access to google if they ask about it. Like I said, I'm not trying to trick them or anything, I just want to know if they can code.

> If you were whiteboard coding and you were telling the candidate "you have a syntax error but I won't tell you where"

I'd hate being on the receiving end of that and would never pull that sort of stunt. The software we use allows both sides to edit the code and I pretty regularly will go in and silently fix syntax errors for a candidate so as not to let a compile/refresh loop get in the way of actually solving the problem.

Like, what does "You have a syntax error" even prove? I'd let you write pure pseudo code so long as it makes sense.

I just don't like "mind games" in interviews.

I still make syntax errors all the time.

I've been writing code for pay for more than 20 years and will forget basic syntax and keywords for a language I wrote hundreds of lines in yesterday without other code to crib off of, IDE assistance, and/or reference material (google, whatever). Like, I'll forget if it's "else if" or "elsif" or "elseif" or "elif" or whatever. Or I'll mix up how to access hash/list members. How to designate a constructor. Stuff like that.

Universities are going to have to adjust. That's really all there is to it. It's not that hard for an exam, since you can put them in a properly designed classroom with human monitors. Homework? Projects? Much more difficult.

The current system worked in 1960. Not in 2023, and probably not since the late 1990s, when technology started to make our evaluation processes obsolete.

Exams used to be a larger part of school grades. In my area, they were devalued and the reason that was given was that exams favored males over females.

I would suggest you ask him. Offer him immunity and ask him. For your peace of mind and so that in the future you can be better prepared.

Can I ask what’s the point of testing students in a scenario where they don’t have access to AI tools? As a software developer I use these tools daily, and if I or my colleagues stopped our productivity would suffer - it sounds like the same genre of thinking that leads to having students write out code by hand because using an IDE is “cheating”.

If it would be possible to take a bottom-tier CS student and turn them into a decent programmer using AI tools, the hiring landscape would be /very/ different. These tools aren’t magic, and to me it sounds like the tests are failing at measuring the students competence if they can so easily be gamed by using an AI/old fashioned googling.

Maybe the solution should be to move towards a style of exam/grading that actually measures the competence of the student in a situation closer to what a professional developer will be in, rather than an old fashioned artificial exam setting?

A person who knows how to code without access to AI tools means when they have access to AI tools their productivity would increase but they will also be able to point out mistakes, if any, in the generated code.

A person who does not know how to code without access to AI tools means they will consistently push bugs generated by AI, affecting the teams productivity, if any.

If this is the case, wouldn't you expect their grades to reflect this as well due to the generated code answering exam questions having errors?

The exam questions are likely simpler than an entire application would be; using AI to write bubble sort is trivial but getting it to understand the full context of your application is much less so.

ChatGPT can solve most LeetCode questions, but I don't think most people here are ready to replace their junior developers with it.

Not on a class that teaches programming to beginners, no. Would you?

I at least partially share this sentiment for hiring interviews. In that setting, the goal of any kind of coding exercise is to evaluate a persons coding skills in a real-world environment. For something like that, it does not make much sense to hobble the developer by artificially limiting their access to coding resources that are normally employed while on the job.

However, I feel like an academic exam is a bit different. The goal of these exams is to evaluate what students have _learned_. Unless the class was about how to find helpful code examples on SO, it does not make much sense to allow AI/internet-searching during the exam.

An evaluation for a course should measure how much of the subjects of the course the students learned. It's not a job success simulator, nor a measure of unrelated skills.

So the students are disallowed or discouraged from learning how to use tools like AI in the courses they take?

Student are neither disallowed or discouraged from learning them. The are not allowed to use them during unrelated test.

Why is the test unrelated to learning methods?

Should we give grade students calculators on their tests that ask for the result of 1+1? Or 10x3?

This follows the same logic as banning calculators when teaching people basic arithmetic. When something is a replacement for your learning, it should be banned, and when it is an augmentation, it should be encouraged.

You may think it's ridiculous, but many people today can't calculate a 20% tip or add two prices together without a calculator because the constant use of calculators has caused their arithmetic skills to atrophy. This seems like a silly gripe, except a lot of other skills are built around these basic ones: for arithmetic, the ability to estimate the cost or time taken to do something is all based on tricks you learn when you are trying to learn mental math. Society is not worse off for this (enough of us still know how to do these things), but many people are poorer, both intellectually and monetarily, due to a lack of arithmetic skills. And no, this lost knowledge of arithmetic is not replaced by a knowledge of higher math - it tends to come with a fear of it.

The same applies with text-generating AI. In terms of writing, if you don't learn to write dumb essays about books, skills like learning to construct an argument are much harder to pick up. For people writing code, learning to slog through writing and debugging a doubly linked list (something ChatGPT can reliably generate for you today) leads you to later being able to slog through debugging B-trees or lock-free queues (which ChatGPT definitely cannot write for you).

I think there is a very compelling argument along these lines for low-level courses to ban AI tools. However, higher-level courses probably should allow students to add them to their repertoire, where they are an aid and not a crutch. This follows how mathematicians and engineers learn to use calculators and computer algebra tools, which seems to work well.

I don’t think it ridiculous at all. I’m not sure if your comment was an argument, a counter-argument or just a comment.

>people today can't calculate a 20% tip

Earlier this week, I spent several minutes talking one of my co-workers through how I 'did math in my head' to figure out my half of the 20% tip we were leaving for lunch. Even something as rudimentary as moving the decimal place over one and doubling the result seemed like wizardry to them.

They aren't dumb. They've literally just never thought one second past reaching for a calculator. Which is kind of scary, because it means they have no way of sanity checking any numbers they come up with.

Most people today can't make fire by hand either because we have tools for it.

It's a good thing that the skills involved in making fire by hand don't transfer to disciplines that matter. The same cannot be said for arithmetic and writing.

I know a person who does interviews for programming positions who asks "what is 20% of 20,000?" The ones who are flummoxed by it are no hires. So are the ones who pull up a calculator app on their phone.

    Maybe the solution should be to move towards a 
    style of exam/grading that actually measures the 
    competence of the student in a situation closer to 
    what a professional developer will be in, rather than 
    an old fashioned artificial exam setting? 

Most "real world" software development work involves understanding existing code and/or choosing solutions.

AI can help you write a linked list, but can it help you know that you need a linked list vs. a queue or a binary tree? Can it help you to make more architecture-y decisions like deciding whether need a document database, a traditional SQL RDBMS, or maybe something else?

I'm not in a teaching role but those are the kinds of things I'd like to see examinations cover -- what tool or data structure are you choosing, and why?

Or, similarly, can the student understand existing code and fix bugs or make performance improvements?

Bottom line, though, is that for most cases, professors should just absolutely accept the reality of AI. They should assume all students are using AI tools. In fact they should probably mandate it in order to assure a level playing field.

And then fail them all because none of them have learned anything.

> Can I ask what’s the point of testing students in a scenario where they don’t have access to AI tools?

It's not just the issue of "what if they didn't have access to ChatGPT?"

Just as an example: ChatGPT is literally the world's biggest bullshitter. It vacuums up vast stores of "data" (note: NOT "knowledge"), and just pulls out words that it thinks could likely follow from previous words given a context.

Quite often is produces silly, or even dangerously wrong, answers. You need to be pretty well skilled in the art to catch it at its bullshit.

It's just like when you hire assistants to do something critical for you. You have to be able to verify their work, or you have to be able to trust them blindly.

Think of it this way. I use a calculator to do less trivial arithmetic. However, knowing how to multiply means I understand what multiplication is rather than just a number that inexplicably appears on the calculator screen.

I've known engineers who did not understand the analytical tools they were using. They misused them constantly.

This is like asking "what's the point of not allowing students to use Google during their Maths exams" or "why bother with English exams when Grammarly exists".

The point is to gain some knowledge yourself. Obviously you'll use new tools and methods later when you eventually get a job, but the point of most courses is to teach you the concepts, not how to command the IDE. Whether you decide to hand-craft assembly code or become professional a Copilot suggestion approver doesn't really matter.

> Can I ask what’s the point of testing students in a scenario where they don’t have access to AI tools?

It's a question of: Are you testing for understanding or ability? Both have their merits, but often the goal is the former. If it were a project, it is generally the latter.

> Can I ask what’s the point of testing students in a scenario where they don’t have access to AI tools?

So that they can tell when the AI tools get things wrong. Which is always.

Basic arithmetic and calculators.

I'm old: when I went to school we weren't allowed calculators "because there'll be a situation where you need to do basic arithmetic and don't have a calculator". I occasionally laugh at this while firing up the calculator app on my phone. We'd have been better off if they'd taught us sign language rather than arithmetic.

Our education system is going to have to adjust to the new reality. Setting an essay task is now the equivalent of learning the times table - utterly redundant (unless you plan on a career writing essays or doing maths).

The most 'fair' math classes (specifically, I'm thinking of algebra and calculus type courses) I have been in, allowed a good in-between. Basically, you were totally allowed to use a Calculator, even one with a built-in CAS, but you -had- to show the in-between steps if you did so on a test.

This was pretty useful in it's own right, as you at least had a way to check your work, and frankly it was extremely useful for helping me solidify knowledge in those fields. [0]

Of course, AI muddles this, since it can explain the steps for you.

That may be the strawman, but that's not the real reason why people ban calculators when you learn arithmetic. The reason is because learning to manipulate numbers in your head abstracts to other things, like learning to estimate and learning to manipulate equations.

Higher-level math classes usually encourage calculators, and graduate-level math classes will often allow Matlab or Mathematica, as long as you can write a short sentence explaining what the computer algebra system is doing. Where I went to school, this was colloquially called "proof by Steven (Wolfram)."

It is a strawman. But I wonder how much better we'd be able to do those other things if we'd been taught to do them with calculators as a tool instead of being forced to do the basic arithmetic in our heads.

If you look at the natural experiment on that in society right now, much worse.

Many people today do math only with a calculator and correspondingly lack any sort of numerical reasoning skills whatsoever, despite having learned both mental math and numerical reasoning in school. That growth of innumeracy coincides with the rise of calculators and computers who do it for you.

Also, students are taught in more advanced classes to use calculators (and later computer algebra systems) as a tool. This comes after they have learned the basics, not as a replacement for the basics.

Writing (hello ChatGPT) and math are like muscles: use it or lose it. If you can't do the basic versions, you can't do the harder stuff either.

See also the ability to read a map...

I'm fascinated by linguistics, and once attempted a degree in Deaf Studies, so yes on the sign language (it's not a language, by the way: it's a class of languages, alongside spoken language, which is another class of languages), but the idea that basic arithmetic is worthless is completely alien to me.

Not worthless, obviously, I was maybe being a tad overblown ;)

But I do think that as we move forward with technological tools for this kind of mental task, that we should embrace them in our education system. If we have to teach every generation from first principles then we won't get them comfortable with the actual tech they'll be using in their lives.

thanks for the clarification on sign language :) One of those things on my bucket list that I keep meaning to learn but never get started on.

So the purpose of academia is teach pupils a subject, that they can the use in real life.

When you're solving the same problems in real life you do have access to all those tools that you banned. Because in real life, if you manage to solve a problem it doesn't matter that much how you solved it.

Sure, if you just Google things and always solve your problems by copy pasting. Your solutions will lack depth, and at some point this will catch up to you.

My point is, cheating is primarily cheating on yourself, and it will catch up to you.

I think this is the framing you should have with regards to cheating.

> Technically, it's possible that he just studied very very hard and his code and grammar improved.

As an early procrastinator (rehabilitated perhaps?) I've had a history of doing very well on finals through cramming.

In your student's case it might very well be more likely they are cheating, but those unlikely but possible students who study extra hard to recover their grades for a final deserve the benefit of the doubt.

How unlikely can academic recovery get? I spent an entire semester completely lost in a second level Macroeconomics course...

... I was able to read and write chapter summaries of a semester and a half of content (about 15 chapters, corresponding to the first level and the second level courses) in a single weekend and aced the exam, with the highest marks in the class.

That was the hardest I've ever crammed in my life and I had trouble speaking in the hours that followed the study session.

I later got a recommendation letter from that prof, totally worth it.

Some students cheat. As an adjunct instructor at the university level, I had similar experiences.

You have to both adjust your expectations and make a significant part of your grading use an in-person one-on-one oral exam or project walkthrough.

It is a huge commitment of time - most professors I know seem to have quietly accepted cheating will be be rewarded because they don't have enough time to verify student performance.

It seems we want to turn universities into McDonalds where we push high numbers, but also pretend they are elite institutions that we can charge premium prices for. This is not going to work well in an age of ever increasing automated intelligence...

Well, lest of course the students sit under the watchful eye of AI in the near future

* * *

By that logic wouldn’t it be ok for grade schoolers to use calculators when they are learning basic math? Sure you can teach someone how to get an answer, but have you actually given them an understanding of the material to be built upon?

* * *

“If we want to ensure kids can do arithmetic without calculators, we can have exams in which there are no calculators.”

Which is exactly what the parent was trying to do with a an exam that forbade the use of AI.