News for hackers. Stuff that matters.
Re:Testing how hard it is to cheat with ChatGPT in interviews by blharr on Wednesday January 31, 2024 @04:09PM (#39209467)
The idea that spotting cheating is obvious is a case of selection bias. You only notice when it's obvious.

Clearly, the person put 0 effort towards cheating (as most cheaters would, to be fair). But slightly adjusting the prompt, or just paraphrasing what ChatGPT is saying, would make the issue much harder to spot.

Re:Testing how hard it is to cheat with ChatGPT in interviews by al_borland on Wednesday January 31, 2024 @04:58PM (#39209987)
Maybe I’m a slow reader, but reading, understanding, and paraphrasing the response seems like it would take enough time to be awkward and obvious as well.

I’m not sure why anyone would want a job they clearly aren’t qualified for.

Re:Testing how hard it is to cheat with ChatGPT in interviews by xboxnolifes on Wednesday January 31, 2024 @05:43PM (#39210463)
> I’m not sure why anyone would want a job they clearly aren’t qualified for.

$$$,$$$

Re:Testing how hard it is to cheat with ChatGPT in interviews by dmoy on Wednesday January 31, 2024 @05:46PM (#39210489)
Well, five moneys at least. They might figure out and fire you before you get to six moneys (but maybe they won't, who knows).
Re:Testing how hard it is to cheat with ChatGPT in interviews by WalterBright on Wednesday January 31, 2024 @09:50PM (#39212278)
It will take 3 to 6 months to determine that a new hire is incompetent, especially if you're required to document their incompetence before firing them.
Re:Testing how hard it is to cheat with ChatGPT in interviews by skeeter2020 on Wednesday January 31, 2024 @11:25PM (#39212699)
I've never had a job without a probation period where you can let someone go without cause within the first 90 days with nothing more than two weeks pay in lieu of notice. It definitely doesn't take 6 months to identify someone who only got their job because they used AI in the interview.
Re:Testing how hard it is to cheat with ChatGPT in interviews by londons_explore on Wednesday January 31, 2024 @06:04PM (#39210690)
As an interviewer, if a candidate can use chatGPT to give a better answer than other candidates, I'm not gonna mark them down for use of chatGPT.

It's a tool, and if they can master it to make it useful, then credit to them.

Alas, ChatGPT seems to be a jack of all trades, but master of none, which is gonna make it hard to pass my interviews which test very specific technical skills.

Re:Testing how hard it is to cheat with ChatGPT in interviews by photonthug on Wednesday January 31, 2024 @07:40PM (#39211561)
> As an interviewer, if a candidate can use chatGPT to give a better answer than other candidates, I'm not gonna mark them down for use of chatGPT.

Tool usage is what separates us from animals and is generally ok where tools are available/expected, but in this case I think you misunderstand which tool we're talking about. The tool involved isn't actually chatGPT, it's more like strategic deception. Consider the structurally similar remark "as a voter, if a candidate can use lies to represent themselves as better than other candidates, I'm not gonna mark them down for use of dishonesty".

The rest of this comment is not directed at you personally at all, but the number of folks in this thread who are extremely eager to make various excuses for dishonesty surprised me. The best one is "if dishonesty works, blame the interviewer". I get the superficial justification here like "should have asked better questions", but OTOH we all want fairly short interview processes, no homework, job-related questions without weird data-structures and algorithms pop-quizes, etc, so what's with the double standards? Hiring/firing is expensive, time-consuming, and tedious, and interviewing is also tedious. No one likes picking up the slack for fake coworkers. No one likes being lied to.

Re:Testing how hard it is to cheat with ChatGPT in interviews by WalterBright on Wednesday January 31, 2024 @09:43PM (#39212242)
> the number of folks in this thread who are extremely eager to make various excuses for dishonesty surprised me

Not me. I see it all the time, online and offline. I suspect they think it confers status on themselves, but what actually happens is honest people wind up shunning them.

Re:Testing how hard it is to cheat with ChatGPT in interviews by verve_rat on Wednesday January 31, 2024 @11:42PM (#39212785)
The problem is that a good interview is only vaguely related to getting a good employee. Anyone can ace and interview and then slack off once they have to job.

If someone aces the interview using an LLM and then does good work using that same LLM then what should the employer or other employees care? The work is getting done, so what's the problem?

Compare a shitty worker to a deceptive one using an LLM. They both passed the interview and in both cases the work isn't being done. How are those two cases different?

Re:Testing how hard it is to cheat with ChatGPT in interviews by recursive on Thursday February 01, 2024 @12:32AM (#39212994)
> Anyone can ace and interview

I'm doubting this quite a bit. If it's so easy to ace an interview, why are there so many bad ones?

Re:Testing how hard it is to cheat with ChatGPT in interviews by verve_rat on Thursday February 01, 2024 @04:13AM (#39214063)
That's not the point I was making. The full quote is:

>Anyone can ace and interview and then slack off once they have to job.

In that a person can pass an interview, get hired, and then not do the job. An interview will never tell you if you will get poor job performance with 100% accuracy.

Re:Testing how hard it is to cheat with ChatGPT in interviews by recursive on Thursday February 01, 2024 @12:06PM (#39218244)
You can't ace an interview and then slack off if you can't ace an interview.

Interviews aren't perfect, but they can still be good filters.

Re:Testing how hard it is to cheat with ChatGPT in interviews by verve_rat on Thursday February 01, 2024 @04:05PM (#39221453)
I don't think you are getting my point. You can totally ace an interview and then slack off. That's it, that's my point. Not the opposite, not something else, just that.
Re:Testing how hard it is to cheat with ChatGPT in interviews by recursive on Thursday February 01, 2024 @05:44PM (#39222538)
Ok. I see. This is theoretically possible. But in practice, I haven't seen it. That's not something I really care about spending effort filtering for in an interview.
Re:Testing how hard it is to cheat with ChatGPT in interviews by photonthug on Thursday February 01, 2024 @12:46AM (#39213057)
> does good work using that same LLM then what should the employer or other employees care?

Maybe I'm wrong, but I find it very hard to believe that anyone thinks the "good work" part here is actually a practical possibility today. Boilerplate generation is fine and certainly possible, and I'm not saying the future won't bring more possibilities. But realistically anyone that is leaning on an LLM more than a little bit for real work today is probably going to commit garbage code that someone else has to find and fix. It's good enough to look like legitimate effort/solutions at first glance, but in the best case it has the effect of tying up actual good faith effort in long code reviews, and turns previously productive and creative individual contributors into full-time teachers or proof-readers. Worst case it slips by and crashes production, or the "peers" of juniors-in-disguise get disgusted with all the hand-holding and just let them break stuff. Or the real contributors quit, and now you have more interviews where you're hoping to not let more fakers slide by.

It's not hard to understand that this is all basically just lies (misrepresented expertise) followed by theft. Theft of both time & cash from coworkers and employers.

It's also theft of confidence and goodwill that affects everyone. If we double the number of engineers because expectations of engineer quality is getting pushed way down, the LLM-fakers won't get to keep enjoying the same salary they scammed their way into for very long. And if they actually learn to code better, their improved skills will be drowned out by other fakers! If we as an industry don't want homework, 15 interviews per job, strong insistence on FOSS portfolio, lowered wages, and lowered quality of life at work.. low-effort DDoS both in interviews or in code-reviews should concern everyone.

Re:Testing how hard it is to cheat with ChatGPT in interviews by verve_rat on Thursday February 01, 2024 @04:19AM (#39214103)
The premise of my comment was: if a person passes an interview using some tool and then uses that same tool to do the job, then didn't the interview work?

You found a person (+ tool combo) that can do the job. If that person (+ tool combo) then proceeds to do the job adequately, is there a problem?

If you present a scenario in which a person passes the interview and then doesn't do the job, the you are answering a question I didn't ask.

To you scenario I would respond: the interview wasn't good enough to do its job, the whole point of the interview process is to find people (+ tool combos, if you allow) that can do the job.

Re:Testing how hard it is to cheat with ChatGPT in interviews by wakawaka28 on Thursday February 01, 2024 @01:30AM (#39213263)
Your hypotheticals are all extremely unlikely. People who ace interviews are usually good, and people who lean on stuff like ChatGPT aren't. I'd also rather not have someone dumping massive amounts of ChatGPT output into a good codebase.

>what's the problem?

Using a LLM is akin to copy/pasting code from random places. Sure, copy/paste can be done productively, except ChatGPT output comes completely untested and unseen by intelligent eyes. There are also unsolved copyright infringement issues via training data, and a question as to whether the generated code is even copyrightable as it is the output of a machine.

Re:Testing how hard it is to cheat with ChatGPT in interviews by ipaddr on Thursday February 01, 2024 @02:06AM (#39213450)
People who ace interviews are people with practice. That means you are last in a long line of unsuccessful interviews or the person constantly interviewing and will be leaving you as fast as they came in.

Find someone with a great resume and horrible interview skills. Chances are they have been working for years and are entering the job market for the first time. You are one of the firsts in their interview process. Grab them right away because once they start getting slightly good in the interview process someone will snap them up and realize they got a 10x (whatever it means to that company).

You'll never find that 10x if you are looking at interview performance unless you can compete on price and reputation.

Re:Testing how hard it is to cheat with ChatGPT in interviews by wakawaka28 on Thursday February 01, 2024 @09:33AM (#39216345)
You don't have to guess if someone is entering the job market for the first time. You can just look at their resume.

Interview skill is not some monotonically increasing quantity. It very much depends on how the question hits you and what kind of a day you've had. Also, it somewhat depends on the interviewers' subjective interpretation of what you do. If you're more clever than them, your answer may go over their head and be considered wrong. They might also ask a faulty question and insist it is correct.

I'm not great at interviews myself. My resume is decent, but the big jobs usually boil down to some bs interviews that seem unnecessarily difficult to pass. I don't practice much for them, because I feel like it mostly depends on whether I've answered a similar question before and how I feel that day. I also often get a good start and just run out of time. I've found that sometimes interviews are super hard when the interviewers have written you off, as in you presented poorly in an earlier session and they are done with you. Also, when there is zero intention of hiring you generally, like someone else already got the job in their minds.

Re:Testing how hard it is to cheat with ChatGPT in interviews by dr_kretyn on Thursday February 01, 2024 @02:08AM (#39213461)
> Tool usage is what separates us from animals

It does not. Please ask any LLM for examples of animals that use tools. (My examples: chimpanzees, gorillas, elephants, dolphins, otters...)

Re:Testing how hard it is to cheat with ChatGPT in interviews by photonthug on Thursday February 01, 2024 @12:27PM (#39218545)
I assume you know this is just an expression, and you know that I know that animals indeed use tools. So I'll refer you to community guidelines https://news.ycombinator.com/newsguidelines.html
Re:Testing how hard it is to cheat with ChatGPT in interviews by dr_kretyn on Sunday February 04, 2024 @12:00PM (#39251964)
Apologies, I had no idea that you used this as an "expression". I have only heard this from people who believed it. Also, I don't think this is a good "expression"; at the very least it's misguided, but mainly it's scientifically constraining. As for the guidelines, I think I could quote it back at you.
Re:Testing how hard it is to cheat with ChatGPT in interviews by lucasmullens on Wednesday January 31, 2024 @06:22PM (#39210882)
That's like being okay with a candidate Googling the answer during an interview. Not unheard of, but unusual. It seems hard to test someone's knowledge that way.
Re:Testing how hard it is to cheat with ChatGPT in interviews by krisoft on Wednesday January 31, 2024 @06:54PM (#39211204)
> Not unheard of, but unusual.

At my company we tell people that they should feel free to google or consult references at practical coding challenges.

> It seems hard to test someone's knowledge that way.

I don’t really want to test knowledge but skill. Can you do the thing? At work you will have access to these references so why not during the interview?

Now that doesn’t mean that we are not taking note when you go searching and what you go searching for.

If you told us that you spent the last 8 years of your life working with python and you totally blank on the syntax of how to write a class that is suspicious. If you don’t remember the argument order of some obscure method? Who cares. If you worked in so many languages that you don’t remember if the Lock class in this particular one is reentrant or not and have to look it up? You might even get “bonus points” for saying something like that because it demonstrates a broad interest and attention to detail. (Assuming that using a Lock is reasonable in the situation and so on of course :))

Re:Testing how hard it is to cheat with ChatGPT in interviews by batch12 on Wednesday January 31, 2024 @10:01PM (#39212328)
> I don’t really want to test knowledge but skill

I do want to understand their knowledge. I'll preface questions with the disclaimer that I am not looking for the book definition of a concept, but to understand if the candidate understands the topic and to what depth. I'll often tell them that if they dont know, just say so. I'll start with a simple question and keep digging deeper until either they bottom out or I do.

Re:Testing how hard it is to cheat with ChatGPT in interviews by munk-a on Wednesday January 31, 2024 @07:32PM (#39211489)
The knowledge we're testing is related to how well you can do your job. Work isn't closed book - if you can quickly formulate a good query to grab any missing information off the internet then more power to you. I've worked with extremely stubborn people who were very smart and would spend a week trying to sort out a problem before googling it, there are some limited situations (highly experimental work) where this is valuable but... I no longer work with these people.
Re:Testing how hard it is to cheat with ChatGPT in interviews by londons_explore on Wednesday January 31, 2024 @06:31PM (#39210985)
I'm okay with them googling too. And I tell them that at the start. But if they take ages to lookup the answer when others just know the answer, it's gonna hurt their chances.
Re:Testing how hard it is to cheat with ChatGPT in interviews by gtirloni on Wednesday January 31, 2024 @07:00PM (#39211251)
Sure, they can search it live but you have to assess if they understand what they found. Usually, if they really know their stuff, whatever they find is just gently pushing their working memory to connect the dots and give a decent answer. Otherwise it's pretty easy to ask a follow up question and see a candidate struggle.

It's like in college when you're allowed to take textbooks to an exam. You can bet the professor spent more time crafting questions that you can't answer blindly.

That being said, I think both types of questions have their place in an interview process. You can start with the no searching allowed questions in the beginning to assess real basic knowledge and, once you determine the candidate has some knowledge, you start probing more to see if they can connect the dots, maybe it's architecture decisions and their consequences, maybe it's an unexpected requirement and how they would react, etc.

Re:Testing how hard it is to cheat with ChatGPT in interviews by yunwal on Wednesday January 31, 2024 @06:34PM (#39211019)
Seems much easier than trying to prevent them from using google
Re:Testing how hard it is to cheat with ChatGPT in interviews by oceanplexian on Wednesday January 31, 2024 @08:25PM (#39211849)
I remember the days when Greybeards would look down on me for using Google in my first IT job, they would harp on about how real Sysadmins use man pages and O’Reilly books to solve problems, and if you tried to Google something you were incompetent. I had college professors that told me you can’t use the Internet for research because the Internet is a not a legitimate source of information, only libraries can have real information.

What happened to all those folks? They retired, and turned into Boomers who are now unable to function in society at a basic level and do things like online banking or operate a smartphone.

Re:Testing how hard it is to cheat with ChatGPT in interviews by esafak on Wednesday January 31, 2024 @08:35PM (#39211889)
On the other hand, they knew how their hardware worked. And if LLMs keep improving, we're going to reach the last generation that knew how software worked.
Re:Testing how hard it is to cheat with ChatGPT in interviews by zerbinxx on Thursday February 01, 2024 @01:26AM (#39213250)
We’re pretty close. I’m not sure that 51% of the people I work with understand what DNS is, what a call stack is, what the difference between inheritance and polymorphism is, or what a mutex is
Re:Testing how hard it is to cheat with ChatGPT in interviews by nonrandomstring on Thursday February 01, 2024 @02:16AM (#39213492)
When I'm retired, sitting on the beach with my beer and a good book, please don't come bothing me that your smartphone banking and GPT arse-wiping assistant has gone berserk.

You're on your own matey.

But hey, you'll have Google.

Re:Testing how hard it is to cheat with ChatGPT in interviews by kazinator on Wednesday January 31, 2024 @09:20PM (#39212137)
Would you also say that, "as an interviewer, if a candidate can use their buddy to give a better answer than other candidates, I'm not going to mark them down for using their buddy"?

Even if you don't mind that situation, shouldn't you get buddy's contact information and offer him the job?

Re:Testing how hard it is to cheat with ChatGPT in interviews by smcin on Wednesday January 31, 2024 @10:46PM (#39212542)
That's not a great analogy: you can't do the job with your buddy; whereas some interviewers are ok with, and even expect you to, use GenAI on the job daily. Depends on the interviewer and job expectations.

A better analogy is an interview where you can use a calculator (and not be detected). If the interviewer were only to ask you simple arithmetic questions with numeric answers then sure you'd seem to do well. So interviewers adjust to not doing that.

Re:Testing how hard it is to cheat with ChatGPT in interviews by toast0 on Thursday February 01, 2024 @12:07AM (#39212881)
> you can't do the job with your buddy

I mean, why not? Me and my buddy are a team, hire us both or none of us. Split the salary if you must.

Re:Testing how hard it is to cheat with ChatGPT in interviews by TeMPOraL on Thursday February 01, 2024 @01:21AM (#39213227)
That's called contracting out and it's fine in this way - and so would be use of GPT, in that way and not to beat the interview.
Re:Testing how hard it is to cheat with ChatGPT in interviews by kazinator on Thursday February 01, 2024 @11:53AM (#39218059)
Sure, and also split the dental and other benefits, vacations, and share one building fob, parking pass, cubicle and computer. Also, split the food at the company dinner. :)
Re:Testing how hard it is to cheat with ChatGPT in interviews by thegrim33 on Wednesday January 31, 2024 @10:01PM (#39212332)
To use a slightly more extreme example .. if you were hiring someone to maintain a nuclear power plant, and when you asked them a question about what actions to take to avoid a meltdown, and they had to ask ChatGPT to figure it out, would you really be OK with hiring that person to maintain your nuclear plant? When they don't actually have the knowledge they need to succeed, but instead have to rely on external tools to decide things? If they need to ask ChatGPT for the answer, how do they know if the answer is right? You really think that person, who relies on tools, is just as good of a hire as someone that fully internally knows what they need to know?

Yeah, hiring someone to code a website isn't the same as maintaining a nuclear plant, but it's the same concept of someone that knows their craft vs. someone that needs to rely on tools. There's a major difference in my mind.

Re:Testing how hard it is to cheat with ChatGPT in interviews by hackit2 on Wednesday January 31, 2024 @11:07PM (#39212623)
I hope your statement is hyperbolic because we're all doomed if you expect a person to know how to operate a nuclear power plant. Normally, your testing if they can follow operational procedure that were created by people who designed the power plant in the first place.

Similar it is unreasonable and bordering on negligence to assume a person has the skill set unique to your situation.

Re:Testing how hard it is to cheat with ChatGPT in interviews by anileated on Wednesday January 31, 2024 @11:50PM (#39212816)
Like a university diploma is a signal of being able to learn or at least comply, use of a chatbot is a signal of not bothering enough to learn or comply.

I can see how an applicant who cheats interview with chatbot would later not bother to internalize operation instructions for the job.

Re:Testing how hard it is to cheat with ChatGPT in interviews by zerbinxx on Thursday February 01, 2024 @01:13AM (#39213190)
I’d like to believe the common line that chat GPT is “just a tool” and that it can actually be used to learn/comply just as much as a university degree can be obtained by mere compliance or demonstration of learning (or merely giving the appearance of such).

My experience with Chat GPT ranges from “it’s really good for rapidly getting a bearing with a certain topic” to “it’s a woeful substitute for independently developing a nuanced understanding of a given topic.” It tends to do an OK with programming and a very poor job with critical theory.

Re:Testing how hard it is to cheat with ChatGPT in interviews by anileated on Thursday February 01, 2024 @02:57AM (#39213696)
> a university degree can be obtained by mere compliance or demonstration of learning

Exactly. It “only” shows you can & willing to at least understand the requirements, internalize them well enough, and comply with them. It shows your capability of understanding & working together with other humans.

Which is key.

In my impression, almost always the knowledge you receive at the uni is not really pertinent to any actual job, and anyone can have PhD level understanding of a subject without having finished high school.

It is the capability of understanding and working in a system that matters.

Similarly with a chatbot. Using it to game interviews in ways described does not mean candidate is stupid, or something like that. It is, though, a negative signal of one’s willingness and intrinsic motivation to do things like internalizing job responsibilities & procedures, or just simply behave in good faith.

Mental capacity to do mundane things is often important when it comes to, say, maintaining a nuclear reactor.

> just a tool

> it’s really good for rapidly getting a bearing with a certain topic

Perhaps. Personally I prefer using Google, so that I at least know who wrote what and why rather than completely outsourcing this to an anonymous team of data engineers at ClosedAI or whatnot, but if it is efficient to get some knowledge then why not?

It’s using it to blatantly cheat and do the key part for you where it becomes questionable.

Re:Testing how hard it is to cheat with ChatGPT in interviews by hackit2 on Thursday February 01, 2024 @01:46AM (#39213330)
ChatGPT like all transformers (language models) depends on how well you prime the model as it can only predict the next series of tokens over a finite probability space (the dimensions it was trained on) , it is up to you as the prompt creator to prime that model so it can be used as a foundation for further reasoning.

Normally people who get bad results from it would also get similar results if they asked a domain expert. Similarly different knowledge domains use a different corpus of text for their core axioms/premises, so if you don't know the domain area or those keywords your not going to be able to prime the model to get anything meaningful from it.

Re:Testing how hard it is to cheat with ChatGPT in interviews by TeMPOraL on Thursday February 01, 2024 @01:14AM (#39213197)
If the job at your nuclear power plant were so simple you only needed the employee to follow operational procedures, then you'd be better off scripting it instead, or training a monkey.

Consider e.g. being a pilot, or a surgeon - two other occupations known for their extensive use of operational procedures today. People in those jobs are not being hired for their ability to stick to a checklist, but rather for their ability to understand reasons behind it, and function without it. I.e. the procedures are an important operational aid, not the driver.

Contrast with stereotypical bureaucrats who only follow procedures and get confused if asked something not covered by them.

Now, IMHO, the problem here is that, if you're hiring someone who relies on an LLM to function, you're effectively employing that LLM, with its limitations and patterns of behavior. As an employer, you're entitled to at least being made aware of that, as it's you who bears responsibility and liability for fuckups of your hires.

Re:Testing how hard it is to cheat with ChatGPT in interviews by fragmede on Wednesday January 31, 2024 @11:05PM (#39212618)
in terms of tools, I absolutely want the nuclear power plant engineer to use a wrench and pliars and tongs and a forklift and a machine while wearing a lead lined safety suit instead of wandering over to the reactor in a t-shirt to pull out the control rods with their bare hands. You could be Edward Teller and know everything there is to know about nuclear physics but you're not getting anywhere without tools.

to your point though, a person needs both. all of one and none of the other is useless. You don't want someone who doesn't know what they're doing to play around disabling safety systems so you don't get Chernobyl, but for the everyday crud website you can just hire the coding monkey at a reduced cost.

Re:Testing how hard it is to cheat with ChatGPT in interviews by wakawaka28 on Thursday February 01, 2024 @01:21AM (#39213226)
>As an interviewer, if a candidate can use chatGPT to give a better answer than other candidates, I'm not gonna mark them down for use of chatGPT.

I think that makes you an incompetent interviewer, unless your questions are too hard for ChatGPT. In any case, solving the question without ChatGPT is more impressive than using it. Just like most other tools, like search engines or IDEs.

Re:Testing how hard it is to cheat with ChatGPT in interviews by Aeolun on Wednesday January 31, 2024 @07:19PM (#39211386)
> I’m not sure why anyone would want a job they clearly aren’t qualified for.

Well, I suck at interviewing and/or leetcode questions, but have so far done perfectly fine in any actual position.

I can totally see how you’d resort to ChatGPT to give the interviewers their desired robotic answers after 3 months of failing to pass an interview the conventional way.

Re:Testing how hard it is to cheat with ChatGPT in interviews by munk-a on Wednesday January 31, 2024 @07:29PM (#39211470)
I understand that you have no control over who you're interviewing with but... if you're a good fit and the interviewer leaves thinking you're a terrible fit that's a sign of a bad interviewer. Obviously there are non-proficiency things you can do to skew that perception (bad hygiene, late, obviously disinterested) but a good interviewer (especially one used to working with developers) should be good at getting by all the social awkwardness to evaluate your problem solving.

And yes, most large companies have terrible interviewers.

Re:Testing how hard it is to cheat with ChatGPT in interviews by aydyn on Wednesday January 31, 2024 @07:56PM (#39211675)
Yes agreed that it is a problem with interviewers, but in practice all the responsibility falls on the interviewee.

I've never once seen an interviewer getting better in any company I've worked for. What happens is they just move onto the next interviewee.

Re:Testing how hard it is to cheat with ChatGPT in interviews by Aeolun on Wednesday January 31, 2024 @10:23PM (#39212433)
I refuse to believe that all the interviewers I had over the course of 6 months were all terrible. It must be something about the process that is pathologically broken (especially when getting hired at larger companies)
Re:Testing how hard it is to cheat with ChatGPT in interviews by verve_rat on Wednesday January 31, 2024 @11:51PM (#39212820)
I mean... if the interview process is even a little broken then doesn't that mean that over time worse and worse interviewers will get hired, making for worse and worse interviews meaning that worse and worse interviewers get hired...
Re:Testing how hard it is to cheat with ChatGPT in interviews by Swizec on Wednesday January 31, 2024 @08:37PM (#39211902)
> give the interviewers their desired robotic answers

As someone who has interviewed a lot of people – robotic answers are specifically not what I (we?) look for. The difference between hands-on experience and book knowledge is exactly what we're trying to tease out.

It's very obvious when someone is reciting answers from a book or google or youtube or whatever vs. when they have actually done the thing before.

For the record: ChatGPT is very good and the answers it gives are exactly the kind of answers that people with book knowledge would give. High level, directionally correct, soft on specifics.

I mostly interview seniors, you obviously wouldn't expect experience from an entry-level candidate. Those interviews are different.

Re:Testing how hard it is to cheat with ChatGPT in interviews by davidthewatson on Wednesday January 31, 2024 @10:08PM (#39212368)
That resonates. Me too!

Here's Pew's Janna Anderson in 2015:

    "Algorithms are taking over much of the human work of hiring humans. And, unless they are programmed to seek out currently undervalued and difficult-to-track factors, they may tend to find that the more robot-like a human is the best she or he will be at doing most jobs. So, it could be that the robots are most likely to hire the most robotic humans."
https://medium.com/@jannaq/the-robot-takeover-is-already-her...

I find the whole gamified system to be bizarre and disheartening no matter which side of the table you're on.

To me, looking at modern tech interviewing is like comparing the gold standard OCEAN and the emergent HEXACO in personality surveys. Take the former on a bad day and it may leave the test taker feeling bad about themselves. The latter, much kinder and gentler in messaging around strengths and weaknesses.

That "by design" quality strikes me as missing from the entire tech interview system. If it weren't broken, this would not be a 7-year conversation updated yesterday:

https://github.com/poteto/hiring-without-whiteboards

Re:Testing how hard it is to cheat with ChatGPT in interviews by notnaut on Wednesday January 31, 2024 @07:06PM (#39211293)
Consider a situation where you’re applying for a job that you’re 50% qualified for and then using chatgpt to cheat on the interview. Would be much more difficult to catch is my guess.
Re:Testing how hard it is to cheat with ChatGPT in interviews by throwaway2037 on Wednesday January 31, 2024 @11:57PM (#39212843)
This is an interesting thought experiment.

If you slide from 50% to 99%, how do people feel about using ChatGPT? What is more honest: Many people here were hired when they were less than 100% qualified, and did very well in their new role. It has happened to me more than once.

Re:Testing how hard it is to cheat with ChatGPT in interviews by bradlys on Wednesday January 31, 2024 @07:19PM (#39211387)
Just because I can’t recite rabin-karp off the top of my head or some suffix tree with LCA shit for some leetcode question about palindromes doesn’t mean I’m unqualified to do the work of an engineer.

I’ve gone public, been acquired by Google, and scaled solutions to tens of millions of users. I’m probably overqualified for your CRUD app.

Re:Testing how hard it is to cheat with ChatGPT in interviews by Atotalnoob on Wednesday January 31, 2024 @07:50PM (#39211644)
We are interviewing for a principal at my work, and the directions from management are to find someone really good.

Instead of tearing into their experience, my coworker is asking what you would use the X class for.

Drives me fucking nuts. Who memorized all the parts of a random, mostly unused .NET class.

I asked the coworker afterwards if he ever used said class, dude said no.

How is that a fair question if it isn’t even used here?

Re:Testing how hard it is to cheat with ChatGPT in interviews by stevenally on Wednesday January 31, 2024 @10:20PM (#39212417)
Exactly. I would never interview for a job, be humiliated by a moron.

Luckily I have the contacts and experience to never have to.

Re:Testing how hard it is to cheat with ChatGPT in interviews by redsoundbanner on Thursday February 01, 2024 @12:10AM (#39212890)
>I’m not sure why anyone would want a job they clearly aren’t qualified for.

Easy. They have nothing to lose because the jobs they are qualified for don't even pay enough to survive. You probably could have figured this out yourself.

Re:Testing how hard it is to cheat with ChatGPT in interviews by TeMPOraL on Thursday February 01, 2024 @02:57AM (#39213705)
> I’m not sure why anyone would want a job they clearly aren’t qualified for.

Money, obviously.

Software jobs in particular are magic in this way - the pay is way above the average, and performance metrics are so poorly defined that one can coast for months doing nothing before anyone starts suspecting anything. Years, even, in a large company, if one's lucky. 80% of the trick is landing the first gig, 15% is lasting long enough to be able to use it as a foundation of your CV, and then 5% is to keep sailing on.

No, really. There's nothing surprising about unqualified people applying for software companies. If one's fine with freeloading, then I can't think of easier money.

(And to be fair, I'd say it's 10% of freeloaders, 10% of hard workers, and in between, there's a whole spectrum of varying skills and time and mental makeups, the lower half of that is kind of unqualified but not really dishonest.)

Re:Testing how hard it is to cheat with ChatGPT in interviews by irrational on Wednesday January 31, 2024 @04:59PM (#39209995)
We will have to start studying people's eyes to see if they are moving as if reading text.
Re:Testing how hard it is to cheat with ChatGPT in interviews by Fetiorin on Wednesday January 31, 2024 @05:32PM (#39210356)
There is already an app from Nvidia that simulates constant eye contact with the camera
Re:Testing how hard it is to cheat with ChatGPT in interviews by mtlmtlmtlmtl on Wednesday January 31, 2024 @07:21PM (#39211402)
Fantastic, now my social anxiety will cripple me in video chats too.
Re:Testing how hard it is to cheat with ChatGPT in interviews by wellthisisgreat on Wednesday January 31, 2024 @06:06PM (#39210726)
It’s really uncanny - it give you an intense, unblinking stare
Re:Testing how hard it is to cheat with ChatGPT in interviews by dghlsakjg on Wednesday January 31, 2024 @11:32PM (#39212739)
Apple ha it built in to iPhones now too. Center Stage is the name of the feature, but I think it is only available on FaceTime calls
Re:Testing how hard it is to cheat with ChatGPT in interviews by klausa on Thursday February 01, 2024 @01:31AM (#39213269)
That's not what Center Stage is.

Center Stage is a feature where a device uses an ultra-wide camera, and then is supposed to track your _face_ as you move and shift around it it's field of view.

I find it most useful for FaceTime calls on Apple TV, where you can leave your phone near the TV, and it will automatically frame you sitting on the couch and will follow you as you shift around, etc.

There is a similar feature to what you're describing for FaceTime, but I don't think it has any cutesy name.

Re:Testing how hard it is to cheat with ChatGPT in interviews by dghlsakjg on Thursday February 01, 2024 @09:21PM (#39224162)
You’re right, center stage is not the name of the feature. The feature does exist though, I believe it is called Eye Contact.

https://appleinsider.com/articles/20/09/25/how-to-use-faceti...

Re:Testing how hard it is to cheat with ChatGPT in interviews by throwaway2037 on Thursday February 01, 2024 @12:01AM (#39212861)
I heard a rumor that Apple FaceTime does something similar, but I could not find any definitive evidence of it. Does anyone know more about it?
Re:Testing how hard it is to cheat with ChatGPT in interviews by dylan604 on Wednesday January 31, 2024 @06:00PM (#39210643)
Just shrink the width of the text area being read from. It's really easy to not look like you are sitting in the front row of the theater reading the opening text in Star Wars. If an actor on live TV can do it, you can too
Re:Testing how hard it is to cheat with ChatGPT in interviews by ThrowawayR2 on Wednesday January 31, 2024 @05:20PM (#39210234)
I predict that that will be followed shortly by a mysterious sharp increase in applicants claiming to have nystagmus (https://en.wikipedia.org/wiki/Nystagmus), which causes random involuntary eye movements, but without any medical documentation.
Re:Testing how hard it is to cheat with ChatGPT in interviews by Volundr on Wednesday January 31, 2024 @05:33PM (#39210365)
What's interesting is this wouldn't necessarily imply cheating. That doesn't sound like an issue I'd necessarily draw attention to under normal circumstances, but if I knew interviewers were likely to be paying close attention to my eye movements I certainly would.
Re:Testing how hard it is to cheat with ChatGPT in interviews by ThrowawayR2 on Wednesday January 31, 2024 @05:55PM (#39210583)
Yes, exactly. I have nystagmus myself because of an underlying medical condition that causes other vision problems and it's depressing that interviewers might think it's reason for suspicion.
Re:Testing how hard it is to cheat with ChatGPT in interviews by bitzun on Wednesday January 31, 2024 @07:07PM (#39211300)
I've been a pain in the ass in quite a few feedback sessions when people brought up a candidate not making "enough" eye contact. Usually I mention that they could be treading into infringing on a protected class and they shut up.
Re:Testing how hard it is to cheat with ChatGPT in interviews by jurynulifcation on Wednesday January 31, 2024 @05:26PM (#39210307)
why wouldn't a cheater just pipe a generative audio model through a small earbud? like that one villain from season 3 of westworld
Re:Testing how hard it is to cheat with ChatGPT in interviews by ozim on Wednesday January 31, 2024 @06:35PM (#39211028)
If someone is smart enough to go away with it is enough that I know but it doesn’t bother me much- I don’t mind.

Had an interview take home assignment done by GPT and it was easy to spot after seeing dozens of solutions. Downside for the guy was - it didn’t work.

Re:Testing how hard it is to cheat with ChatGPT in interviews by godelski on Wednesday January 31, 2024 @07:02PM (#39211262)
You probably should because this is a person demonstrating that they can and will hack metrics rather than consider what you're proxying.
Re:Testing how hard it is to cheat with ChatGPT in interviews by ozim on Thursday February 01, 2024 @02:48AM (#39213664)
In my current setting it doesn’t work like that.

We have a small team of developers and you cannot hack metrics. You build and deliver what is in requirements or not.

If you don’t deliver we don’t even have to have a discussion because team reviews code, tests features and gives feedback quickly if someone is slacking.

Re:Testing how hard it is to cheat with ChatGPT in interviews by godelski on Thursday February 01, 2024 @12:13PM (#39218346)
All metrics can be hacked.
Re:Testing how hard it is to cheat with ChatGPT in interviews by ozim on Thursday February 01, 2024 @04:50PM (#39221953)
Not going to write too much but I think this is best reply:

https://xkcd.com/810/

Re:Testing how hard it is to cheat with ChatGPT in interviews by godelski on Thursday February 01, 2024 @05:09PM (#39222141)
Well you did say that your metrics "cannot be hacked."

The XKCD one can actually be easily hacked. Just spam the system and rate every comment as helpful. Classifier learns to accept everything. There's dozens of way to hack this one and undermine the actual goal. But it is a comic, and it is funny. Doesn't need to be realistic.

Re:Testing how hard it is to cheat with ChatGPT in interviews by twic on Wednesday January 31, 2024 @07:32PM (#39211496)
What's your evidence for that claim?
Re:Testing how hard it is to cheat with ChatGPT in interviews by johnnyanmac on Wednesday January 31, 2024 @10:18PM (#39212409)
I mean, at some point if they go through so much effort to hide their cheating they probably have attained some mastery in the process. Kinda like how some friends in high school would try and sneak in note cards on a test but they probably spent so much time prepping them that they coulda gotten an A or B regardless.

It's also why it's kinda annoying to do live interviewing trivia questions. Can I immediately answer what a partial template specialization is? Probably not, I never used them. Can I google it in 2 minutes and summarize it as as way for (often c++) template classes to bound some of the template arguments to values or pointers? Well, I just did. Should that cost me the interview? That's pretty much what I do on the job.

Re:Testing how hard it is to cheat with ChatGPT in interviews by throwaway2037 on Thursday February 01, 2024 @12:00AM (#39212859)
I am a polyglot: Perl, Python, C, C++, Java, C#, etc. Not experts at all, but I can do fine with an existing code base. What is it about C++-heavy interviews that always regress to trivia? And asking about rarely used features? It is a bother. And rarely does the person asking the trivia have any depth whatsoever in other languages. It is my biggest gripe with "C++ people". For many, they have a hammer and everything looks like a nail. Yes, "Java enterprise people" were the same in 2005-ish.