Next semester, we plan to start doing "interviews" of students in which they explain their own code to us.

It obviously doesn't scale, so we'll use random sampling.

As a TA we did something similar: we asked them to self grade their own homework using a provided rubric, and then we spot checked 1/4 of the students (without replacement) to punish lying about what grade you deserve. We didn’t punish for a few disagreements over the rubric, but if it was blatant we checked their assignments every time in the future (and told them). I think if it was bad enough we could have reported them.

This saved a bunch of time on actually grading assignments and made us write a very clear and unambiguous rubric (which required a very clear homework) and also demonstrated to the students that grading was not arbitrary.

> It obviously doesn't scale

Peer teaching/grading potentially scale out.

Several universities [1] scale out personalized instruction and interactive grading by hiring students from previous cohorts and paying them either in course credit (taking a "course" that involves teaching students in the current cohort) or at a low rate (possibly subsidized by financial aid) comparable to other on-campus student jobs.

[1] "Scaling Introductory Courses Using Undergraduate Teaching Assistants", SIGCSE '17, https://doi.org/10.1145/3017680.3017694

(I want a browser extension linking doi to google scholar btw)

How do you justify the fact that only some of the students get the pleasure of an in-person grilling? Or, am I completely misunderstanding the process you're going to be using?

In my plan, each student is interviewed at least once. Ideally more than once by the same teacher, so the teacher can get to know them a little better, spot areas where the student needs more help, etc.

There's still a scaling problem, but I think it makes the ~200 student classes we have now more feasible than 100% autograding. I also like the other commenter's suggestion of coming back to interview certain students each time, if they need it.

Is this about pleasure or about measuring knowledge?

A lot of stuff you learn and the way you learn it isn't necessarily pleasant, but frequently you still have to do it and you really discover 20 years later why it was needed.

No, it's about why only a subset of students get singled out for extra scrutiny, literally arbitrarily, as the selection procedure itself is defined as "random sampling."

random sampling is an effective method for inferring the same information about the larger population that is being measured in the smaller sample, to a certain degree of confidence based on the sample size and known distribution of what is being measured. These concepts are fundamental to statistics.

In college, viva-voce is a significant part of non-theory exams. It’s another matter it was not run well by many colleges but I always loved those chit chat sessions with some of the good professors. Some professors treat it like a boring Q&A which reduces its effectiveness.

I think you might be who the top response is responding to. You seem to have inside knowledge that saving money is the top priority without considering any real-world resource constraints.

The top response is the one that brought the constraint of "scale" into this discussion, and that's what I'm addressing. Maybe you should bring your objections to them rather than to me. "Real-world resource constraints" is just a euphemism for "wanting to save money" in this case. I'll edit it to clarify that I mean the same.

And I'm not passing judgement on the choice made, nor saying the constraints aren't there, nor saying anyone should do anything different. I'm just pointing out that the scalability constraint will affect the test possibilities, which will affect the quality of the measurement. Feel free to disagree with this all you want.

EDIT: Also, I do happen to have some inside knowledge by having worked in higher education for about a decade, starting in the mid 00s. Coincidentally, most of my work was on cost-saving measurements, designing a few algorithms that allowed universities to reduce their teacher headcount (first at a university, then at a software vendor), so yes, the #1 goal there was saving money. But I don't think having done this affects my answer, nor I do think that I deserve special treatment. I'm merely answering to a chain of comments.

Wanting to save money also falls under availability of staffing trained to do this. Considerations of if the massive increase of expense and diversion of people from other economic endeavors is worthwhile.

Good hunch, but in my experience, availability of trained staff was never really an issue in practice. Hiring well trained university faculty was always purely an economical problem. Universities often already have a trained surplus of faculty employees working in a highly reduced capacity. Especially in the last 10-15 years where distance learning became commonplace, a lot of faculty was replaced by low-paid part-time quasi-teachers, which would be more than happy to be offered a permanent position. To further demonstrate that this is an economical problem: those quasi-teachers often have different job titles other than "teacher", depending on the jurisdiction, in order to evade laws and evade the reach of (often very powerful) faculty unions.