News for hackers. Stuff that matters.
Re:Testing how hard it is to cheat with ChatGPT in interviews by leeny on Wednesday January 31, 2024 @03:46PM (#39209153)
I'm not the author (perhaps he'll chime in as well), but I'm the CEO of interviewing.io (we're the ones who ran the experiment).

I think it depends on whether the interviewer has agreed to make the interview "open book". Looking up stuff on Stack Overflow during the interview can be OK or can be cheating, depending on the constraints.

In this experiment, the interviews were not "open book". That said, I am personally in favor of open book interviews.

Re:Testing how hard it is to cheat with ChatGPT in interviews by michael_mroczka on Wednesday January 31, 2024 @03:48PM (#39209184)
I AM the author, and I also am in favor of "open book" interviews. I'm not against ChatGPT use in interviews, but if you're doing it secretly in an interview that clearly is meant to be "closed book," I think it's fair to say you're cheating.
Re:Testing how hard it is to cheat with ChatGPT in interviews by jacques_chester on Wednesday January 31, 2024 @04:52PM (#39209917)
> I also am in favor of "open book" interviews.

I recall reading an interviewing.io blog post[0] in which the dominant considerations interviewers weighed were (my interpretation):

(1) Did they solve the problem optimally? (2) How fluid was their coding?

With "communication" turning out to be basically worthless for predicting hire/no-hire decisions.

Perception of coding fluidity seems like it would be affected by how often the candidate stops and looks up things like library functions or obscure syntax.

For that reason I've been investing time in committing a lot of library functions to memory, so they instantly flow from my fingers rather than spending a minute looking it up.

It's dumb that I need to do this, but I don't make the rules. I'm just at the bottom of the information cascade that led to how things are done now.

[0] https://interviewing.io/blog/does-communication-matter-in-te...

Re:Testing how hard it is to cheat with ChatGPT in interviews by madeofpalk on Wednesday January 31, 2024 @05:09PM (#39210116)
Companies that optimise for memorising obscure stdlib functions don't seem like great places to work.
Re:Testing how hard it is to cheat with ChatGPT in interviews by spongebobstoes on Wednesday January 31, 2024 @09:06PM (#39212051)
Occasionally looking something up is normal, but if you don't know how to append to a list in Python or iterate over a vector in C++ then you probably are not currently writing much code in those languages. That's a signal by itself, and one that is too often a negative.
Re:Testing how hard it is to cheat with ChatGPT in interviews by mark38848 on Thursday February 01, 2024 @03:28AM (#39213849)
Why? Maybe you just use good programming languages instead.
Re:Testing how hard it is to cheat with ChatGPT in interviews by doix on Thursday February 01, 2024 @03:50AM (#39213955)
I'm not saying you're wrong, but that reasoning is why I have to prepare for interviews. I'm _really_ bad at remembering that stuff, I think it really depends on how you think while programming.

I've got an abstraction that I think in that then needs to be translated to code. e.g. if I want to append to a list, I think "push to list", regardless of language/framework/whatever. Then somehow my hands will translate that automatically to code in the language I'm working in. If I'm not in my usual editing environment, that magic just sort of breaks, and I just look incompetent.

It's not a huge deal, but I have to actively sit and memorize that stuff before an interview. Usually by writing it out on paper or writing it in some foreign editor that I'm not used too.

It probably wouldn't be _too_ bad today, because I just sit and write Typescript everyday, but when I was switching between perl/ruby/python/tcl/kotlin/javascript/bash/csh/lisp my brain was basically mush. I couldn't tell you how to do any basic operation in any language.

Re:Testing how hard it is to cheat with ChatGPT in interviews by KronisLV on Thursday February 01, 2024 @09:29AM (#39216287)
> I've got an abstraction that I think in that then needs to be translated to code. e.g. if I want to append to a list, I think "push to list", regardless of language/framework/whatever. Then somehow my hands will translate that automatically to code in the language I'm working in. If I'm not in my usual editing environment, that magic just sort of breaks, and I just look incompetent.

Thank you for putting all of that into words that I can look at and go: that's exactly how my brain works!

I don't think everyone has that issue or can understand it. I think in more abstract terms (the problem and how to solve it) and more often than not the syntax is just an implementation detail that I prefer to increasingly outsource to my tooling.

Re:Testing how hard it is to cheat with ChatGPT in interviews by mewpmewp2 on Thursday February 01, 2024 @03:43AM (#39213914)
I don't think companies intentionally optimise for that. It is just the fluid way of writing code that impresses whoever is watching you code.

But I do think most of what I use has grown into my muscle memory naturally, rather than memorizing anything.

Re:Testing how hard it is to cheat with ChatGPT in interviews by gabrieledarrigo on Wednesday January 31, 2024 @04:20PM (#39209569)
> but if you're doing it secretly in an interview that clearly is meant to be "closed book," I think it's fair to say you're cheating.

I would argue that is the opposite: it's fair to say that the interview is a cheat.

Re:Testing how hard it is to cheat with ChatGPT in interviews by randomdata on Wednesday January 31, 2024 @04:17PM (#39209545)
> an interview that clearly is meant to be "closed book,"

I am not sure that is clear. It seems the expectation was not "closed book", but "never opened a book before, not even in the past":

"It's tough to determine if the candidate breezed through the question because they're actually good or if they've heard this question before."

Clearly the interviewers were looking not for knowledge, but for uncanny ability. How well was that communicated to the interviewees?

It is not cheating if the rules of the game are not defined.

Re:Testing how hard it is to cheat with ChatGPT in interviews by pierat on Wednesday January 31, 2024 @04:12PM (#39209501)
Well that's the rub. There's no way, even for a senior engineer, to know everything. In fact, one of the required skills is "how to ask the question as to elicit the answer in a reasonable amount of time".

The closed-book crap can stay closed in the universities and schools demanding a regurgitation of mostly-right knowledge.

Now... The skill of asking the right Qs also directly intersects with LLMs, and how to discern good/bad responses.

But hiding it? Yeah, probably not a good fit.

Re:Testing how hard it is to cheat with ChatGPT in interviews by vinni2 on Wednesday January 31, 2024 @04:47PM (#39209864)
> The closed-book crap can stay closed in the universities and schools demanding a regurgitation of mostly-right knowledge.

I work at a university and most of our exams are open book or project based. You probably want to update your image of universities.

Re:Testing how hard it is to cheat with ChatGPT in interviews by ziddoap on Wednesday January 31, 2024 @06:21PM (#39210875)
>You probably want to update your image of universities.

I have a close friend who is a prof at a university and most exams remain closed book.

You probably want to update your image of universities.

Or, perhaps, we can agree that it depends on the university, the subject, etc. and blanket statements based on single anecdotes are silly?

Re:Testing how hard it is to cheat with ChatGPT in interviews by BriggyDwiggs42 on Wednesday January 31, 2024 @04:02PM (#39209384)
Given your knowledge of the subject, do you think leetcode-type questions are meaningfully able to appraise an employee’s performance in a production environment? I’ve always thought it was basically unrelated beyond testing basic coding experience.
Re:Testing how hard it is to cheat with ChatGPT in interviews by michael_mroczka on Wednesday January 31, 2024 @05:19PM (#39210229)
The short answer is, "Yes." They are very flawed, but one of the most reliable ways to avoid "bad" hires.

The longer answer is: Fundamentally, you need to address the fact that there exist a huge number of people in this industry declaring they have masters degrees/phds or years of industry experience, but when pressed they can't write even the simplest of functions.

While we called it out explicitly, some folks seem to miss that "Custom" questions are still fundamentally DS&A leetcode-style questions. I completely agree that "leetcode style" interviews are flawed, but most people don't have a better answer for this problem that still guarantees the person you're hiring actually can code.

We are optimizing for coders that make good choices quickly, and if they can code efficient code to toy CS problems, then you at least guarantee that 1) they can actually code and 2) they can code simple things quickly. Non-coding interviews allow you to hire people who can't do these basic things and therefore guarantee their performance is worse in a production environment.

Re:Testing how hard it is to cheat with ChatGPT in interviews by consp on Thursday February 01, 2024 @03:23AM (#39213819)
You forget where you lose good hires since leetcode questions are pathetic and demeaning for anyone with experience.
Re:Testing how hard it is to cheat with ChatGPT in interviews by trgdr on Thursday February 01, 2024 @05:51AM (#39214622)
Keep it civil, dude. This is a pretty reductive take that is not obviously true and adds little to the discussion.
Re:Testing how hard it is to cheat with ChatGPT in interviews by michael_mroczka on Thursday February 01, 2024 @10:55AM (#39217252)
Let's keep it civil, my friend. I totally get the frustration with leetcode-style questions and acknowledge that they aren't perfect. They can indeed overlook the diverse strengths that experienced engineers bring to the table. My perspective is that understanding the basics, like the efficiency of different data structures, is crucial, not just for interviews but for making informed decisions in our work.

I'm not advocating for spot tests on complex algorithms without context. However, I believe a conversation about fundamental concepts like big-O notation reflects on one's approach to problem-solving and software design. It's not about dismissing anyone's experience or capability but ensuring a solid foundation that benefits all aspects of engineering work.

I understand senior engineers being out of practice and not being able to derive things like topological sort on the fly, but an inability to talk about the basics of big-O or the simplest of data structures is more commonly a red flag than "rust." If they never learned the material, that's a red flag. If they learned the material and they've been building software for the last decade without taking the basics of the material into account while coding, then that is a red flag as well.

Again, happy to admit that this is a flawed approach that will lose some great engineers, but most engineers that have your line of thinking are ones I actively avoid hiring. These concepts are practical, available to learn for free, and easy to understand. If an engineer feels that testing these concepts is beneath them, then they definitely aren't a good fit for any team I'm on. Big-O is to software engineers what Ohm's Law is to electricians. Imagine a world where electricians thought it was demeaning to talk about Ohm's law in an interview. ¯\_(ツ)_/¯

Re:Testing how hard it is to cheat with ChatGPT in interviews by mewpmewp2 on Thursday February 01, 2024 @01:09PM (#39219142)
> Big-O is to software engineers what Ohm's Law is to electricians.

Really? Most of the work that exists is CRUD, and I've seen people who grinded leetcode algorithms, do 1+N SQL queries which is way, way worse, than forgetting what kind of time complexity a sorting algorithm has that you never implement yourself anyway.

At the same time these leetcode grinders will over engineer a UI solution to make sure they use the correct data structures to avoid looping 300 times in the browser at the cost of readability writing 30 lines of code instead of 4.

I don't remember a single time where I've used Big O in practice.

I ask myself - what will happen with performance if there's N amount of traffic or data, but I don't think of it in terms of Big O. I consider perhaps a time it takes or load it takes at upper limit numerically.

N might in some cases be terrible, and N times N performance in other cases totally fine.

I think actually thinking in terms of Big O might make you worse engineer as you start to obsess over the wrong things.

I prefer someone who's spending all that time on building things instead. This time of leetcode grinding and memorising data structures, algorithms could be used on coding side projects.

You learn way more and more practical things when you build something.

Re:Testing how hard it is to cheat with ChatGPT in interviews by mewpmewp2 on Thursday February 01, 2024 @03:48AM (#39213940)
I usually use tasks where they would build in an existing codebase like setup. Frontend or backend and they have to add features to or find bugs.

The whole algorithm and leetcode thing just seems so off and unpractical to me.

Then once they have done the exercise there are so many practical things we can discuss about the existing code. Peformance, quality, etc, etc.

Re:Testing how hard it is to cheat with ChatGPT in interviews by tomp on Thursday February 01, 2024 @06:22AM (#39214799)
It's important to note here that leetcode questions doesn't mean "LeetCode Hard" where the only solution is an obscure algorithm that is on average never used in production code (except deep in libraries you didn't write).

Leetcode questions can be fairly basic algorithmic/data structures questions, like "how do you store an ordered collection with O(1) lookup (ordered dict, i.e. hashtable + list)" or "check if this list contains duplicates" (use a set, not a list!), or "calculate the square root of this number" (bisecion, i.e. looping).

These are not production code questions, but the benefit is that they're easy to explain and understand in the limited time you have available for an interview.

Unfortunately there are more than expected number of people who fail these basic questions. You really don't want to hire them.

Re:Testing how hard it is to cheat with ChatGPT in interviews by blitzar on Thursday February 01, 2024 @05:05AM (#39214361)
I have been part of a few interview tests that were proudly "open book". However, if you are going to read the context and tasks then build 20 spreadsheets in 20 minutes you have 0 time to google something.