For example, I once copy pasted an OSS code base into ChatGPT and all hallucinations went away. I had to limit to the files relevant to my questions, due to the context window limitation, but worked really well.
Providing hints is a bit tricky, however. Often times, students fail the test case for unexpected reasons, and our hints actually mislead them. I've had students come to me (as a TA) saying that the test case is wrong because they take into account the hint.
Naturally, we could just give better test cases / give a broad disclaimer, which we try to. We do the latter, but the former is tricky, especially when we are trying to come up with novel problems.
The problem with first year CS students is that, unlike math, they’re allowed to have a huge range of possible prior experience in the subject. Some have zero programming experience whatsoever, while others started programming in diapers and competed in numerous programming contests (and even won) before enrolling.
Yes, we do offer an advanced version of the intro course that is far more challenging and would be appropriate for many of those advanced students. But the advanced course is optional so no student can be forced to take it. Many of the advanced students opt to take the standard intro course in the hope that it would boost their grades.
So the course is dealing with (essentially) a strongly bimodal distribution of student ability and the professors don’t want to give the advanced students a free ride. That means the course is actually somewhat challenging for all but the most elite programmers and therefore extremely challenging for total beginners.
What this meant was that Monday office hours (assignments were due Tuesday mornings) typically had several hundred students (out of a total of about 1800) scrambling for help from a dozen TAs. Of course, my school only has a tiny fraction of the funding of Harvard, but there you go.
I doubt that Harvard would want to spend proportionally more on TAs for their course (which would mean more TAs than students), so even though their course may be way more “well-funded” than ours, it’s not at all so relative to endowment.
HN readers: universities that teach a so-called "How to Design Programs" curriculum [1] have discovered a partial solution to this problem. Emphasizing a design process in a functional language removes the advantage that students with experience but not a deep understanding gain from knowing how to cobble things together the way they learned in high school AP CS A.
I'm not saying it's the only route, but it works well and develops important skills that are (ime) not learned by tinkering.
[1]: https://htdp.org [2]: https://course.ccs.neu.edu/cs2500/labs.html
We even hand-mark their design recipes in the course! The students just do them after the fact.
They also don’t want to allow the advanced students to “test out” of the intro course, as some schools do, because the advanced students may be highly skilled at programming but totally lacking in introductory CS knowledge.
Source: Am a high school programmer
Firstly, a big part of the allure of CS50 is the fact that it's one big ol' nerdy computer party and you're invited!
Secondly, and I'm not trying to sell the Harvard mystique, but considering that Harvard's acceptance rate is 4%, the students handle brick-wall challenges gracefully.
Additionally, they don't necessarily want students that are going to do poorly in CS50 to pursue computer science at Harvard, so I'm sure it also serves as an effective bouncer for all of the students who haven't decided on their concentration deciding that they'll do CS because tech is sexy right now. The lectures look pretty straightforward, but we had to write code involving basic data structures and memory management in C on our paper midterm exams, with no reference, administered in a lecture hall. It's probably tough for kids that are used to acing everything, but it's a lot easier to find out you don't like CS in CS50 than it is to find out after choking on discrete math or a compiler class.
And as my first-year expository writing professor said after assigning like 70 pages of close-reading Kant for a weekly assignment, "Will it be difficult? Yes. Reading Kant is like running in sand. But that's why you decided to go to Harvard."
It's been a while since I was an undergrad, and I didn't study CS when I was (I was in math -- we didn't even get the unit test to tell us we were wrong lol). I understand auto-grading like this is very common in lower level CS classes, but wouldn't it be possible to write these "test suites" in such a way that they could provide some hints on exactly what was wrong? I have to imagine, based on my experience as a graduate TA in math, that undergrads as a whole make mistakes that typically fall into certain classes, and that those classes could be reflected or captured in the pattern of which "unit tests" end up failing. It doesn't strike me as terribly difficult to analyze these patterns of failures, after running the course at least once, and come up with useful hints as feedback.
Does nobody do this? Or, perhaps more to the point, does nobody ask their TAs to do this?
> Totally legit philosophical debates aside, I think this is a great use of the technology. It's a very, very well-funded class (bigger dedicated full-time year-round staff than some entire departments,) so I'm sure it's not a half-assed effort.
I agree broadly with this. If you're going to do this experiment, in principle, a class like Harvard CS-50 sounds like the right place to start, for the reasons you've listed.
However, if this chatbot is just an equivalent of ChatGPT with the GPT-4 model and some additional training/fine tuning, I don't think the technology is quite there yet. We've seen a lot of examples already of how GPT-x has a tendency to give "confidently incorrect" answers, and that's the absolute last thing you want to be telling beginners. I've seen answers that could plausibly fool experts, yet be wrong enough as to be useless once you try to verify the details. The danger here is that beginners may not recognize this, and may not be able to verify the details.
That said, I think there is a future for this sort of technology in precisely this type of instructional environment. The "confidently incorrect" problem is something that can be mitigated with fine-tuning, proper prompt engineering, and other techniques, but, to my knowledge, can't be eliminated if the system is essentially a bare text interface into an LLM. I think the work being done with web search is a possible direction here, since I've seen GPT-4 essentially give up and go search the web when pressed for details or when the actual answer would become too complex. For a restricted domain like CS50, maybe even some kind of formally encoded knowledge base would help it (similar to how I mentioned analyzing patterns of failures to provided hints).
What I do know is that the gigantic "hall-o-TAs" is absolutely not a great user experience for undergrad learners, so anything that offers the potential for improvement on that UX is definitely something I'd like to see pursued and developed.