One of my courses, which only had about 8 students and two instructors, decided to do an oral examination, which ended up being basically a very in-depth, one on one conversation about the course material and based on the expectations set in the syllabus (so, no surprises).
While obviously not practical for large rosters, this was by far the best exam format that I have ever done in my many, many years of schooling. I'm sure not everyone would prefer it, but the students unanimously agreed to try it (wouldn't have done it that way otherwise), and it was just so great. It was not at all like an oral thesis defense, which was what I was a little worried about.
Professors are also given quite a lot of flexibility in their grading. My mother had a fun story about a professor she had in college - a professor of a really hard math class who wanted to save on exam time. He announced "exam will be hard. Anyone willing to settle for a D, bring your report cards forward, I will mark them D and you can leave. No exam.". Some people came forward, got their Ds marked, and left. Once the door closed, he said "Anyone willing to accept a C, please come forward". Some did. After the door closed there, he announced to the remaining smiling students expecting easy As/Bs: "I'll see you all for the exam tomorrow 8 am".
No way this could happen in USA.
I've heard that story many decades ago in the form of a joke. It may have started from a professor who genuinely didn't care about failing students but did care about identifying the best.
This really is the best case, but as you note it was 8 students so quite manageable.
It requires a little skill on the part of the examiner, but you can quickly find out how much material the student knows with much higher accuracy than other exam formats, in my opinion.
One of the skills needed is to be able to make it conversational-feeling and reduce the anxiety of students. You can often tell when a student mostly knows what is going on but has misstated or misremembered something, and guide them around the place they got stuck.
One unfortunate thing is that poorly done, orals can be very uneven.
And while it was possible to de-anonymise, the academics were all in support of blind grading so why would they?
The only exceptions were projects where everyone was assigned a different topic, and graded presentations.
Of course, this is simpler in STEM subjects - it's not like you can guess someone's race or gender from their switching power supply design. Subjects that prize in-class participation and lengthy essays would probably be harder to blind-grade effectively.
If you are teaching a 4th yr/masters mixed class of 11 by yourself, you pretty much get to know who is who whether you want to or not. I suppose avoiding handwriting can help if it's appropriate (e.g., won't work on a math course) but I suspect you'll know everyones style by then anyway.
It certainly could, but it also certainly couldn't. I imagine being confidently incorrect is likely to produce a worse result than being unconfidently(?) incorrect, for example.
Similarly, a less confident speaker may end up spending more effort justifying their answers, which could better expose their knowledge.
I think it would depend quite a bit on the examiner in this case. Some people may even be simply biased against particularly confident speakers, particularly considering the relative positions of the speaker and examiner.
If the examiner isn't much more knowledgeable about a topic than their students, then something else has gone wrong.
I'm delighted to hear that it went so well, and I am a believer in the idea. I have seen, from time to time, oral thesis defenses become rather tense and difficult, and think that things go better in proportion to the preparation of both student and examiner. Any general observations about what worked, for those contemplating giving exams in this fashion?
The examiner (one of my two instructors) had a list of questions/topics that we had to get through, but the specifics and flow were natural and spontaneous rather than artificial and forced.
What made the conversation good was that the examiner discussed points that I raised, raised points that I didn't, asked my opinion on e.g. real world implications of theories or conclusions that could be drawn if this or that were true, et cetera. This made it two-way and engaging. While he did not give me any answers, of course, when it was clear that I could bring up and sufficiently discuss a topic, then he would go into details, which would trigger even more detailed responses from me, and so forth. In this way I think he was able to probe the depths of my understanding while not needing to employ a one-sided question/answer format.
I think that it is difficult to bullshit a topic in depth with someone else who knows what they are talking about, so an oral exam probably does not need to be a hardcore opposition like a thesis defense might be.
The allotted time was maybe 30 or 45 minutes (I don't quite recall), after which the examiner would tell you what grade they would recommend. If it was a grade with distinction, then of course you would say thanks and be done. If it was less than distinction then you could request another 10 or 15 minutes of further examination to try to show your mastery. (I passed with distinction, so I didn't go through that part, but I assume it would have been a continuation? But maybe it would get a little more intense if you were trying to improve your performance at the last minute? I don't know.)
I was able to see current and past residence data, which I suppose could have helped me verify the student's identity ("what post code[0] do you currently live at?"), but I tried not to look at that information when possible.
Lecturers also had access to this system (it dealt with some submission data), so I guess they may have had access to view residency data?
This is just an educated guess based on my experience, though.
[0]: For American readers, think ZIP code
Crazy stuff.
My own experience of making up those cheat sheets is the same as yours, the effort put in to create them is an excellent teaching tool in itself, as you're studying and distilling all the stuff you don't know well, and by the time you're done, you hardly have to refer to it in the exam anyway.
For physics, they'd just give us a page with all the formulas but with no context aside from the letters in them, so if you knew roughly what it should look like, you could find it and you're not at so much risk of making a basic error under pressure.
This raised eyebrows of the TA's proctoring the exam, but like, if you don't know the stuff, a page isn't going to help you at that point.
She's a fully online professor, though, so maybe that's the difference? Different dynamic from the traditional, in-person classes professors?
I also don't think anything but her quizzes are multiple choice? Everything else is essay format, participation (discussion posts in class forum), and a HUGE paper, so there's really nothing to cheat on, aside from plagiarism, and ... there are ways of catching that.
For advanced degrees, though, I'm not so sure. I expect someone which a masters or Ph.D. to actually be an expert in the subject matter, not just someone who is really good at figuring out how to solve problems. A big part of that is being able to internalize the information so well they you in effect become a resource that could be used by an undergraduate student. This internalization goes beyond rote memorization, but memorization is a big part of it too.
Just a disclaimer, though - I do not have an advanced degree, so maybe I expect too much from those who do? A big reason why I have no interest in pursuing one is that many people with a masters degree I find have little expertise to show for it, they could have just as easily learned the same information by self-study or being fortunate enough to find interesting work. (the hiring landscape is a separate topic)
I still cannot "remember" how to perform a discrete-wavelet-transform from memory (my honor's thesis) but i found myself digging into it a couple of months ago (just fiddling on pet projects). An hour on googling got me on the track. Point is there is so much even those with advanced degrees have to know and having to retain it all in memory all the time is both infeasible and wasteful. Yet expecting kids/young adults to do this is truly hypocritical and disingenuous.
2. Master’s degree courses follow all kinds of schemas for examination. From oral exams, to in person no extra material allowed, to open book, and take home. Being on both sides (taking the exams and creating them) I can say that it really doesn’t matter if you allow people to take the exam home and collaborate among each other. These types of exams are designed to really test a deep understanding and ability of the material. There have been exam questions where an entire class of >20 students are not able to solve it. People that are really good are able to have a shot at it and maybe make some progress, and that differentiates the good from the exceptional.
3. Nowadays undergraduate degrees are a commodity so it feels they need to make sure only the good students get one. But in the end it doesn’t really matter, since most employers (exceptions of course) will want to see how much value the candidate add, which does not correlate with having a degree or good grades
Of course you can, making up results is a time-honored tradition!
It does take a certain amount of skill to do it believably, though.... ;-)
At graduate level, there is too much stuff. You know the outline, you know where to find stuff. But you don't need to know everything exactly. If you forgot one passage from a definition, that should not cause a failure. Because in 'real Mathematics' you get to look at references.
A nice trick for open book is to make it time-infeasible to just look up everything. But allow students access to the materials incase they have a brainfart.
Which is a load of dingos kidneys.
Gate keeping.
All the good doctors I've ever been to have a wall of reference material and use it, even if only to show the patient, but it's there and accessible.
And besides, it's not like practising medical doctors don't make heaps of mistakes.
The number one cause of complications in a medical setting is medical intervention, so it could be argued doctors should be using more reference material and not relying on their over worked brains.
I lived with a orthopedic surgeon doing a specialization in pediatric orthopedia.
He was very very very good based on my understanding of the places he had worked, the offers he had to undertake his specialization, the change in operation waiting times and his model of how he worked within the hospital, and that immediately after returning to his home country he was head of the newly created orthopedic pediatrics ward.
His description of how he would approach a surgery was straight-forward:
- he would review the surgical referral, his notes from his patient intake
- he would review appropriate notes and documentation from the hospital on the allowed surgeries allowed, including reference texts on that particular surgery
- he would create a full plan for entering, performing, and exiting the operation
- (surgery)
- he would review the plan, add notes on how it actually went
So it's not like he woke up, walked into a surgical room, and ad-libbed it.
As an anecdote, he said he liked B-students for his surgical teams because they had to work hard to get there.
I also was good friends with a very nerdy emergency room doctor. He offered the anecdote that he was "called out" for using Google. His justification was that "once you throw in the appropriate medical terms, all you're getting is the latest research on treatments--and I was attempting to offer the best and latest in proven care". He also made the several changes to the shifts at the ER, including using time-sheet and patient data to show that patient outcomes suffered during longer doctor shift rotations. His anecdote on his schooling; "I was a bad student, and I'm a poor doctor, but my job is to get you stable until we can intervene properly". He funded a startup and wrote (in Java) a medical EMR system.
We need more doctors, if that means changing the culture within the profession so they're more inclined to use those reference material that would be good thing.
Overall I think an oral exam run by a skilled examiner is the best of all worlds, but it isn't practical for a section of 500 undergraduate students. It's quite doable for a seminar or manageable up to say 10-15 students, becomes difficult after that.
In some European universities, the outcome is such a foregone conclusion that the candidate's family is laying out the buffet, peeling the cling film off the plates, and unboxing the champagne bottles as the "defense" begins.
In the UK however, I've never seen this. It's generally a small room with candidate, an internal examiner, an external examiner from another university who is an expert in the field, and a convenor to record the minutes of the examination. The outcome is by far from a foregone conclusion.
A good student who is an expert in their field, is well-read and up to date on their work and the surrounding literature will perform well and have little to fear. Someone who hasn't written their own thesis, or didn't really have an understanding of the area, and thus isn't really an expert, will have a very unpleasant time, and will likely be failed, or be sent away with major corrections to be completed to the satisfaction of the examiners, possibly including a full oral re-examination.
America is leaking again.
Ok, from a terminology point of view that's fair - it isn't the same everywhere by any means.
However, most if not all of the graduate programs I know if internationally have something roughly equivalent, whether they are comps or prelims or qualifying or whatever.
The basic idea is that a department (and university, generally) has an interest in maintaining the quality of their programs, and one way to do that is to make sure that your students never leave with glaring holes in their background. The best way to do this is some sort of comprehensive evaluation, and the time to do it is at or near the beginning of a program - otherwise there is no time to address deficiencies.
The common factor among all UK PhDs I know of is that there is a rigorous viva at the end, where the outcome for the sudent is not a foregone conclusion. Despite the shorter overall duration of the PhD (~3 to 4 years typically), the oral examination can (rightly) cover material far beyond the scope of your thesis - if you are an expert in your field, you will be able to have a knowledgeable and informed discussion as a peer with your external examiner, who will be a recognised expert in the field. I'm definitely a big believer in the importance of being able to have a well-informed discussion around the area of your work, and actually found the whole viva process very enjoyable and cordial - a nice chat about the wider field, my and the examiners' own previous work, some debate of the merits of different approaches, and then onto a run-through of the thesis, chapter-by-chapter, skipping any pages where there were no points for discussion or contention.
Unlike European vivas though, there's no family or friends, no champagne corks being popped mid-defense, and no foregone conclusion of the outcome. I've been at European vivas with the family of the candidate preparing the celebratory buffet at the back of the auditorium while the questioning continues!
While the UK model is probably preferable in a lot of ways, this sounds much more enjoyable :)
At least that's the theory - I don't know if in practice it holds up; most of the grad students and later I knew from that system came from oxbridge which has a number of quirks.
My experience of it was that if you are genuinely knowledgeable and approaching being an expert in your field, it is an enjoyable experience, and just like having a (longer than usual, but not uncomfortably long) conversation with someone about a topic that you both share a deep interest in. There is nothing to worry about, as you can have a nice discussion about an interesting topic, and share interesting ideas etc.
I'm not sure if it's a lower bar for grad as such - I think it's got fewer "formal" requirements, and far more informal requirements. The most common way to "fail" is to simply not submit the thesis.
My impression was that someone failing their viva was seen as a serious failure on the part of their supervisor, who should not have let them submit without being more prepared.
That matches my limited experience with UK, although I've seen north american ones without a foregone conclusion - seems really dependent on institution.
> The most common way to "fail" is to simply not submit the thesis.
I think that's true everywhere, lot's of people just don't complete. I've heard somewhere that the "ABD" (all but dissertation) rate in US is nearly 50% at 10 years, if you go across all disciplines.
The two are pretty much equivalent, but with different names.
Open book is also fine in this case because without a base of memorized understanding an open book isn't that helpful anyway.
If you don't know the answer to a question and you look it up, that's not cheating, it's research, or at least referring to reference material, which is always permitted in real life. Except for some contrived or rare circumstances someone will now point out. Even the Apollo 11 crew had support.
Any professional in any industry is allowed to say: "I don't know" and "I'll get back to you" and "let me look that up".
I don't know about you, but I tend to want my pilots, surgeons, and lawyers to tend towards self-competent, and I don't think those circumstances are contrived or rare. It's not to say that they can't ask for help before or during flight/operation/trial -- and in fact asking for help and crew-resource management is important for those fields -- but not all fields are like software engineering or aerospace where there may be dozens or hundreds of people that can help you in a timely fashion if you have completely no idea what you're doing. I'd prefer pilots, surgeons, and lawyers that didn't cheat their way into their jobs.
Real life isn't always kind enough to only give problems that someone has already solved and published somewhere you can find.
Knowing how to find or determine the current correct answers yourself is often more useful in the long run than memorizing what amounts to trivia. This is especially true when it comes to open ended and quickly changing fields...
This doesn't solve the problem of a student having an expert sitting off camera, feeding the student the answers.
Employers, students and society as a whole have all moved on; they want assessment to demonstrate that students can do what the course has taught them (known in the jargon as “alignment”), not memorise a bunch of facts that they can regurgitate on demand.
Which is the hard problem these solutions try to address, because even if the correct student is sitting in front of the computer, that doesn't mean he isn't just proxying the questions and answers to someone else (possibly paid).
Brute force memorization, however, are subject to decays and constant forgetting. It is astonishingly difficult to memorize a bunch of meaningless numbers unless you taken the time to do a mnemonic for it.
So they'll keep asking stupid questions as if the student was a database being queried. It's no wonder that people can solve them by querying actual knowledge bases! Schools won't accept that: they are selling degrees and they don't want to make it too easy for students to pass their courses. So they come up with the absolutely ridiculous notion of cheating.
The whole thing's so phony and artificial it's mindboggling. I don't think it's even possible to get a high quality education anymore. Maybe if a student is lucky enough to have a one-on-one mentorship.
Giving students different exams (selections of slightly and subtly different tasks from a task pool), not showing task names, and mixing up the order of tasks for each student seems to work quite nicely.
The increased overhead to communicate which exact answer they need, finding out which other student has the exact same task, etc., has worked really well for us. Of course, you have to have it in a way where they do not have much time left over to shoulder this overhead.
Edit: So students are going from "What's the answer for 5" to sharing the topic of the task, the task description (there may be a negation hidden in there), and the constants and other students needing to compare. Maybe they aren't even at this task-type in their exam yet, as the order is mixed.
Edit2: We weren't really watching the video stream. It was just to discourage students actually sitting side-by-side, which would decrease the communication overhead drastically.
As for alternative options, most my classes used open book paired with more limited time restraint, possibilities of questioning students if anything looked suspicious immediately after the exam, and making the tests harder and scaling afterwards. Although there'll always be bad apples, I'm thankful many of my profs has adapted very well to help the students learn.
Student's post: https://www.reddit.com/r/UBC/comments/hgiiu1/midterm_started...
Proctorio CEO's comment (u/artfulhacker) got deleted back is on removeddit, Linkletter is also here defending the student as well: https://www.removeddit.com/r/UBC/comments/hgiiu1/midterm_sta... https://www.reddit.com/r/UBC/comments/hhbowl/proctorio_ceo_m...
They've moved to Lockdown Browser without the recording, and to Zoom proctoring. In my opinion, neither are particularly effective measures against cheating, and I'm sure they are trivially bypassable.
The effect of these tools being phased out is that exams now must be harder or less student friendly. Typical practice in some of my courses has been to not allow students to go back to answered questions, while giving large amounts of questions with insufficient time. The exams are scaled, but I can imagine people doing worse in this kind of stress.