All that said, it's an awful system and I ragequit the take-home midway through. It's buggy, slow, lacks useful hotkeys, and basically requires you to become accustomed to a wiped Linux box without any of the tools you would typically use. Sure, you can install them - but it's a timed assessment, for crying out loud.
Finally, I suppose nothing stops you from logging in with your credentials on your authorized machine and then physically handing that machine to a paid agent.
Not everyone opts for that (maybe their local environment is messy, or they only have a locked down work device, etc), but I always make it an option.
I've learned many things over the years watching other experts in their local environments as an interviewer. New packages, new shortcuts, new helper apps, ...
I'm a Spacemacs user and I had to do the remote interview for my current gig in Google Docs. That was a bit frustrating.
Who said anything about overseas? Hire your classmate to sit down at your own PC to do it. That's the Hard Problem.
Just the knowledge that another human being was proctoring the exam made the idea of attempting to cheat pretty far fetched. My body language would have given it away, and it wouldn't be easy to transition smoothly from the interactive setup to some kind of pre-recorded loop without it being easily detected. The kind of person who could successfully pull that off... well, could also probably pass the test on their own merits.
So, the first step is just to actually take the test. The cheating bit would be having someone monitoring what your doing via a reasonably hi-res hidden camera/etc pointed at your screen/etc. Then all you need is a channel for that person to communicate with you. Be that a tiny bluetooth hearing aid in one ear, a phone/screen taped hidden on your leg/etc or to your existing screen. Basically somewhere not visible to any camera's monitoring you.
So, when the person monitoring you detects you have answered something incorrectly they send you something to the effect "question #5's correct answer is Y because A,B,C". Then later you get a moment of insight and go back and fix the problem. In theory, unless your completely clueless the difference between a 2 and a 4 is just a couple questions so this would only have to happen a couple times during the whole test. Again, unless your completely clueless, if the proctor notices you fixing an incorrect answer and asks "why did you just change your answer" you give them something reasonable sounding. Heck for some of them, it might not even have to be the right answer, its completely possible for someone to get the right answer the wrong way in a multiple choice section. AKA, "I guessed on that one the first time and A & B seemed far fetched so I just picked C over D, but then later I questioned my assumption and decided it was B & C that seemed far fetched so I picked the remaining answer which was D rather than A".
Anyway, I'm not sure the screen is needed. For a test that is 100% multiple choice, a buzzer in your shoe would be enough if it went off whenever you selected a wrong answer, and then pulsed a code to indicate the right answer.
If you can pre-record stuff like that without knowing what the sequence of requested events will be, I'm impressed.
Save the 1:1 interviews for students whose original program was great but whose modification went down in flames and you'll likely be talking to your outsourcers, plus identifying students who maybe could use some extra help even if they did everything themselves.
I am also skeptical you could find appropriate trivial changes to request. Students, and particularly beginner students, may write very unconventional code that may not be as easily modified as a well-thought-out solution.
What about 1-on-1 interviews? Ask people the questions face-to-face, ask them to talk you through their answers. That's how advanced degrees like PhDs are assessed.
What we care about is that they can demonstrate they understand the problem and solution. The code is just the starting point for that conversation.
We have hired people who submitted failing solutions because they were able to think through the problem on the spot when we told them it failed and asked why.
Dijkstra exclusively gave oral final exams on a board. Allegedly, he stopped one of his former students after the first question and told him something to the effect of "you clearly know what you are doing by how you answered that, so you'll get an A, but your handwriting is atrocious." He spent the rest of the exam time making the student practice penmanship.
I'm unconvinced that oral exams provide the most secure and consistent form of assessment. I'm not even sure how to evaluate such a claim, though, so it may have to stand as an opinion.
I don't think there is an off-the-shelf product for cheaters, but it's only a question of time. Especially now that I mentioned it.
They probably failed even by the simple arithmetic that the time required to make a minimally good faith effort at applying a grading rubric is greater than the time a teaching assistant is paid to "assist" in the grading.
Your teaching assistants almost certainly scaled back their time grading your assignments to fit their allotted hours. If any of those intro classes fulfilled a writing credit you can probably start with available undergrad writing credit requirements, figure out how many hours your TA got paid, and look back at the number of students for a given TA to figure out just how much money you and/or your parents lost on the deal.
If you consider college tuition to be the money you pay to be educated, you're definitely right. Personally I think of college as something like a licensing or certification.
I learned twice as much at my part time programming job than I did in class (but classes where instrumental in getting me through technical interviews). But, once I had my degree I instantly had multiple 100k+/yr offers in a moderate COL area. For me, that is a good deal (but still a huge hassle).
If I think of class theater in an education that led me to 100k+/yr, that is a good deal.
If I think of class theater in a certification that led me to 100k+/yr, that is a good deal.
That it is a good deal doesn't change the the fact that class theater is a shady practice.
It's also detrimental. But since we don't have the tech to do speed runs against the versions of ourselves who took non-shady intro classes it can be easy to shrug.
And I can't do that without proper assessments!
While some might argue my approach is too aggressive, I report every account that messages me asking to cheat to Wyzant as well as the appropriate organization. My goal is to make it simply not worth the attempt to cheat (obviously only if they explicitly ask; I don't report when it's a possible miscommunication). Just about all of them have had their accounts deactivated.
If technology access could be assumed, I don't think it would be unreasonable to ask students to upload a picture of themselves with their photo ID, as well as some sort of validity check, like a unique code only provided to the device they're taking the exam on. Obviously, that can't be implemented because of equity issues. Ideally, a secured assessment would be done on something similar to Pearson Vue, ProctorU, or the multitude of other online proctoring services. Unfortunately, that's not an option due to both scale and technology access (not everyone has a compatible device, I'm sure many students are taking the exam on their phones).
Someone who wrote the code themselves will know right away and someone who purchased a project won’t have enough time to sift through the code to figure out the answer.
Also, anyone remember the story of the programmer who subcontracted his work to some offshore freelancer for a fraction of his salary? He just sat at his cubicle goofing off all day while the freelancer did all the work.
You propose a lab report, with example data, and I actually really like that idea, but I think it would fail on that last score. Looking back, AP chem is a lot of inorganic reactions (lots of memorization, some principles), acid base work, some organic chem, some lab techniques. It’s fairly scattered. A single (or several) lab reports would not capture all of it, especially given that chemistry doesn’t work perfectly predictably even with a post-graduate understanding. There’s a reason basic science lab courses do boring experiments, it’s because those work reliably.
Anyway, I like the idea, and want to hear more on how to do it, but a lab report would have to be supplemented by something else to really evaluate understanding in the sciences.
On the APCSP exam, there's actually very few points for your code itself. You get a point for identifying an abstraction and a point for identifying an algorithm, but the rest of the points (there are 8 total) are assigned based on your written responses. Analytical questions could be added to each exam–broad enough that no two students should have too much overlap, but narrow enough that it's clear what is expected of a student response.
From there, a few points could be assigned for the quality of the designed experiment, but a majority would be assigned based on the quality of the written response questions.
AP Physics follows a similar model on the regular paper exam–you have to prove some theory and design a complete experiment to do so. I believe it also included one light analysis question.
Lock someone in a room with some reagents, and expect them to produce extracted/cleaned output products of a certain testable purity and mass after N time. Make the process of getting from the reagents to the final product really involved, requiring at least one of every kind of basic reaction/setup taught in the course. (NileRed's "aspirin to tylenol" sequence might be a good example challenge: https://www.youtube.com/watch?v=b0Ejuew2riA)
You know this is "the practical approach", because this is essentially how gangs qualify their chemists. "If you can really make [illegal drug], then make us some! Without doing anything that'll get the police called! In four hours!"
That sort of practicum may be a reasonable test, but could give very different results than the normal written exam.
> If I were to design these assessments, I'd have it be an untimed, synthesis-based assessment...
Yes, but that takes more resources and has more variability among reviewers. It does have the advantages of being more fair for students with time zone issues or closures, and of being durable in the face of temporary website issues. Judging by their decisions, the priorities of the testing agencies seem to have been taking out the cost of the human element of reviewers (multiple choice is easy to grade, synthesis essays need expert reviewers) and the variability of the tests and reviewers to certify their results for schools.
A significant part of taking AP English when I went through was the teacher saying: "Okay, here is the question. Now, what are the likely literature references the AP reader expects you to use? Okay, what are the likely scoring points the AP reader has on their worksheet? Okay, now write to those. Writing anything else is a waste of time."
You could count the number of students out of her class who didn't score a 5 on a single hand.
The English teacher in question was an actual AP Reader as well as an assistant editor of the local newspaper.
This is absolutely hilarious. Did they not realize the concurrency limitations?
Their online tooling has never been great–AP Classroom takes at least 3 tries for me to access, one to log in again which then redirects back to AP Central, one that just redirects back to AP Central, then finally one that takes me to my assignments page.
This certainly isn't an easy problem to solve. CollegeBoard definitely should have designed a more stable system to host their exams on, but as with anything digital, there's always an opportunity for error–even moreso since everyone takes the exam at the same time (which is an entirely different can of worms, with students in Guam taking exams at 1a, 3a, and 5a).
If I were to design these assessments, I'd have it be an untimed, synthesis-based assessment, more similar to AP Computer Science Principles[0], with their Digital Portfolio. Simply having a deadline for all students to upload to a basic tool (which already exists as CB's Digital Portfolio system). Like the current exam, have the submissions sent back to teachers for validation (there's nothing on the current system that validates your identity, simply paste in your AP ID and enter your name/dob, and you're in the system).
I think if a student can write a well-formed essay over a topic, they can get college credit. In science classes, they could design an experiment and simulate a lab report. CS classes can create a program, etc.
[0]: https://apcentral.collegeboard.org/courses/ap-computer-scien...