Consider that this may not be typical.
Source: https://survey.stackoverflow.co/2023/#ai-sentiment-and-usage
To be fair, the number of "Yes" was "just" 43% but that's still a very large amount of developers, not including those who plan to use it.
And I can still use ChatGPT and similar tools for some of what I do. It is a huge force multiplier.
I think it depends on whether the interviewer has agreed to make the interview "open book". Looking up stuff on Stack Overflow during the interview can be OK or can be cheating, depending on the constraints.
In this experiment, the interviews were not "open book". That said, I am personally in favor of open book interviews.
I recall reading an interviewing.io blog post[0] in which the dominant considerations interviewers weighed were (my interpretation):
(1) Did they solve the problem optimally? (2) How fluid was their coding?
With "communication" turning out to be basically worthless for predicting hire/no-hire decisions.
Perception of coding fluidity seems like it would be affected by how often the candidate stops and looks up things like library functions or obscure syntax.
For that reason I've been investing time in committing a lot of library functions to memory, so they instantly flow from my fingers rather than spending a minute looking it up.
It's dumb that I need to do this, but I don't make the rules. I'm just at the bottom of the information cascade that led to how things are done now.
[0] https://interviewing.io/blog/does-communication-matter-in-te...
I've got an abstraction that I think in that then needs to be translated to code. e.g. if I want to append to a list, I think "push to list", regardless of language/framework/whatever. Then somehow my hands will translate that automatically to code in the language I'm working in. If I'm not in my usual editing environment, that magic just sort of breaks, and I just look incompetent.
It's not a huge deal, but I have to actively sit and memorize that stuff before an interview. Usually by writing it out on paper or writing it in some foreign editor that I'm not used too.
It probably wouldn't be _too_ bad today, because I just sit and write Typescript everyday, but when I was switching between perl/ruby/python/tcl/kotlin/javascript/bash/csh/lisp my brain was basically mush. I couldn't tell you how to do any basic operation in any language.
Thank you for putting all of that into words that I can look at and go: that's exactly how my brain works!
I don't think everyone has that issue or can understand it. I think in more abstract terms (the problem and how to solve it) and more often than not the syntax is just an implementation detail that I prefer to increasingly outsource to my tooling.
But I do think most of what I use has grown into my muscle memory naturally, rather than memorizing anything.
I would argue that is the opposite: it's fair to say that the interview is a cheat.
I am not sure that is clear. It seems the expectation was not "closed book", but "never opened a book before, not even in the past":
"It's tough to determine if the candidate breezed through the question because they're actually good or if they've heard this question before."
Clearly the interviewers were looking not for knowledge, but for uncanny ability. How well was that communicated to the interviewees?
It is not cheating if the rules of the game are not defined.
The closed-book crap can stay closed in the universities and schools demanding a regurgitation of mostly-right knowledge.
Now... The skill of asking the right Qs also directly intersects with LLMs, and how to discern good/bad responses.
But hiding it? Yeah, probably not a good fit.
I work at a university and most of our exams are open book or project based. You probably want to update your image of universities.
I have a close friend who is a prof at a university and most exams remain closed book.
You probably want to update your image of universities.
Or, perhaps, we can agree that it depends on the university, the subject, etc. and blanket statements based on single anecdotes are silly?
The longer answer is: Fundamentally, you need to address the fact that there exist a huge number of people in this industry declaring they have masters degrees/phds or years of industry experience, but when pressed they can't write even the simplest of functions.
While we called it out explicitly, some folks seem to miss that "Custom" questions are still fundamentally DS&A leetcode-style questions. I completely agree that "leetcode style" interviews are flawed, but most people don't have a better answer for this problem that still guarantees the person you're hiring actually can code.
We are optimizing for coders that make good choices quickly, and if they can code efficient code to toy CS problems, then you at least guarantee that 1) they can actually code and 2) they can code simple things quickly. Non-coding interviews allow you to hire people who can't do these basic things and therefore guarantee their performance is worse in a production environment.
I'm not advocating for spot tests on complex algorithms without context. However, I believe a conversation about fundamental concepts like big-O notation reflects on one's approach to problem-solving and software design. It's not about dismissing anyone's experience or capability but ensuring a solid foundation that benefits all aspects of engineering work.
I understand senior engineers being out of practice and not being able to derive things like topological sort on the fly, but an inability to talk about the basics of big-O or the simplest of data structures is more commonly a red flag than "rust." If they never learned the material, that's a red flag. If they learned the material and they've been building software for the last decade without taking the basics of the material into account while coding, then that is a red flag as well.
Again, happy to admit that this is a flawed approach that will lose some great engineers, but most engineers that have your line of thinking are ones I actively avoid hiring. These concepts are practical, available to learn for free, and easy to understand. If an engineer feels that testing these concepts is beneath them, then they definitely aren't a good fit for any team I'm on. Big-O is to software engineers what Ohm's Law is to electricians. Imagine a world where electricians thought it was demeaning to talk about Ohm's law in an interview. ¯\_(ツ)_/¯
Really? Most of the work that exists is CRUD, and I've seen people who grinded leetcode algorithms, do 1+N SQL queries which is way, way worse, than forgetting what kind of time complexity a sorting algorithm has that you never implement yourself anyway.
At the same time these leetcode grinders will over engineer a UI solution to make sure they use the correct data structures to avoid looping 300 times in the browser at the cost of readability writing 30 lines of code instead of 4.
I don't remember a single time where I've used Big O in practice.
I ask myself - what will happen with performance if there's N amount of traffic or data, but I don't think of it in terms of Big O. I consider perhaps a time it takes or load it takes at upper limit numerically.
N might in some cases be terrible, and N times N performance in other cases totally fine.
I think actually thinking in terms of Big O might make you worse engineer as you start to obsess over the wrong things.
I prefer someone who's spending all that time on building things instead. This time of leetcode grinding and memorising data structures, algorithms could be used on coding side projects.
You learn way more and more practical things when you build something.
The whole algorithm and leetcode thing just seems so off and unpractical to me.
Then once they have done the exercise there are so many practical things we can discuss about the existing code. Peformance, quality, etc, etc.
Leetcode questions can be fairly basic algorithmic/data structures questions, like "how do you store an ordered collection with O(1) lookup (ordered dict, i.e. hashtable + list)" or "check if this list contains duplicates" (use a set, not a list!), or "calculate the square root of this number" (bisecion, i.e. looping).
These are not production code questions, but the benefit is that they're easy to explain and understand in the limited time you have available for an interview.
Unfortunately there are more than expected number of people who fail these basic questions. You really don't want to hire them.
However, my style of interview is LLM-proof anyways. I have "shop talk" style interviews, where I just chat with the developer for an hour or so about various topics. Makes it very easy to get a sense of their depth, and how interested they are in the job domain.
I am having an interview next week and I hope it will be going like that. They emailed me yesterday with a small coding task. I was supposed to setup a simple server updated website with blazor (horrible name) using (a) Background Worker for the Server side "computation" that updates the page with a random number every X seconds.
Even not having worked with asp.net directly ever it was fairly easy to setup and implement. The complexities are rather well hidden by dotnet. But the use of the BackgroundWorker class seemed weird to me. In fact I implemented it first with a simple timer instead before noticing the ambiguity in the task description. So I implemented it both ways. I think I spend less than an hour on it and thats nice cause it respects my time :)
It was a time sink for sure, but I can't stand coding in front of people because I usually like to sit and reflect. And I had a good amount of time to prepare my thoughts on what I'd built. No hidden surprises, no anxiety. I loved it.
Salesforce had a good interview practice a few years back where they invited you to a meeting. Started a recording, then asked you to keep your microphone and camera on and do several simple programming tasks. It was an "open book," and you could use whatever you wanted, but you just had to show how you got to where you were (and you could only use one monitor so that it was clear what you were looking at at all times). The engineer who met you on the call left after just a couple of minutes, and you could work in peace without having to worry about "entertaining them."
For what it's worth, I think LLMs are very compatible with the style of interview you describe here. For people for whom LLMs have become a part of their workflow, seeing how they interact with them is just one more way to get a sense for how they work and think about things. If it doesn't come up, that's find too! But I don't see any reason not to accept their usage in your interviews.
(But I do think it's key to ask them to screenshare if they're going to go that route, so that you can actually see their interaction to get that signal.)
What I meant was that if you're going to give a live-coding interview, I personally wouldn't accept LLM usage. The reason is that with that style of interview, you're essentially validating one of two things - how much knowledge they have on hand, or how they're able to think in the abstract about problems and reliably solve them. I'm more interested in the latter, so I don't care if they use an external reference for trivia. But LLMs simulate the abstract thinking for you, which means I can't evaluate a candidate's ability to reason their way through a problem.
It actually is very important that you're able to think through problems on your own, sans-llm.
This isn't my experience at all. Or rather, it isn't my experience that people are able to use them to do this, effectively. So if a candidate tries to do that, that's signal.
The point of an interview is to - very imperfectly! - get a sense for how people think and work. If a candidate defers all their thinking to an LLM, that's something I'd like to know about them!
But I've never seen anyone do this in an LLM-allowed interview. Instead, they use it as a more effective version of how people have long used web search and IDE autocomplete, if you allow those. It speeds up the boring parts of interviews where people fumble over silly syntax issues or "what is python's method for xyz thing called" that don't tell me anything about how they think and work in a real setting.
This narrative of "interviews should be like real work" needs to stop. It doesn't make any sense, this is not the goal of an interview.
While the above sounds sarcastic, honestly, it isn't as ridiculous of a thought as it sounds. I don't think being good at DS&A problems guarantees that you're a good coder, but in general, the people who get good at these problems are also great coders. I can think of less than a handful of people who are good at CS problems but bad at actual coding.
Since IT is paid very well, there are lots of people who try to lie their way to get the job. If you are someone on minimum wage, then why not try your luck - if they fire you after 2 months, you probably earned more in those 2 months than in whole year of your shit job.
The point of (software engineering) interviews is to demonstrate how you solve problems. "Type the question into ChatGPT and do what it says" is not the "how" that companies are looking for.
They are probably not looking for that, because LLMs perform poorly with some kind of problems, and you don't want people who rely on them heavily.
This gives you a plan for designing the interview questions.
I'm definitely looking at googling proficiency when interviewing.
Being good at LLMs is about as important - this means being able to tell when you're being confabulated at quickly, knowing what to ask and what not to ask, etc.
Then interviewers should stop setting tasks that require either a) copy and paste answer from leetcode or b) copy and paste answer from chatgpt.
Unfortunately that requires skill and awareness on behalf of the interviewers; who typically served in the leetcode wars and want their employees to go through the same.
In my experience it's mostly been laziness. Grabbing leetcode problems is easy compared to preparing a custom scenario that tests the candidates higher-order skills. God forbid they spend up to a day preparing something that could be reused for the next several years.
After about three interviews it ended up on Leetcode.
The interesting part is what to do when ChatGPT does it wrong, and if the problem is not trivial and an exact solution is not available online, it is usually wrong. Sometimes, not by much, but it takes skill to notice the problem and fix it, either manually or by asking ChatGPT for it.
Same idea as for libraries. One could argue that those using the "sort" function don't know how to write a sorting algorithm, but in real life, unless there are particularly good reasons, rewriting a "sort" function would be crazy, and probably not what you want from your employees.
If you want your candidate not to use the tools at his disposal, you may frame it as a requirement. "on this test, imagine you are working on a sensitive project, you are not allowed to upload any details to a third party service, you can search the internet for generic information, but do as if your development machine has no internet access, so no copy-pasting". Or, for libraries "on this test, imagine we absolutely want to limit dependencies on third party libraries, even if it means sometimes reinventing the wheel, so only <list of libraries> are allowed".
Whether or not it is a security concern depends on what you are working on I guess. So that's what I meant by "I don't know". It may absolutely be, or it may not be considered so, but the idea is that if you are able to do the indirect process, that is not blindly copy-pasting, you are already showing some level of skill.
Unfortunately though, there are lots and lots of interviewers out there that do not give a shit about "how", but rather if you can provide the right answer or not.
Here are three questions, you have 60 minutes. Please provide the optimal solution to at least two of those questions. Now excuse me while I'm listening to a teams/zoom meeting in the background. Good luck, have fun.
I don't, and I don't know anyone working with me thar is using any LLM (we pair). Some tried Copilot at some point and concluded it was useless for our use case.
Not sure on what context one would constantly use LLMs.
I'm not hiring at the level of day to day work, I'm hiring at the level of when things go complex or bad.
To give an example, I'm not going to test a coders ability to write some CRUD application. I'm testing the ability when a junior developers comes with a crazy problem, that they can find the cause and solve it.
If I only test day to day work, you get these kind of developers that keep changing faulty code until it magically works. I don't want those in my team. I want developers that can figure out what is going wrong, understand it, and provide a solution. Do such things happen day to day? No.
And a person that can reason his way out of things instead of trying random stuff, sure as hell can do the same day to day tasks just as well or even better.
Allow them to use the tools, with a screenshare, and adjust the types of tasks you are giving them so that they won't be able to just feed the question to the LLM to give them the completed answer.
Interviews should be consistent with what day to day work actually looks like, which today means constantly using LLMs in some form or another.