This helps us look for good signal around their communication skills, mentoring skills, empathy, reasoning skills and software development skills.
We used this review as a diagnostic - there are no wrong answers since we know they’re not aware of our style guide and conventions.
It worked for us. Maybe it could work for others.
I pose a situation and introduce the v1 code. They look for bugs and propose fixes. Then I add on a new feature, and they look for more bugs. I've maybe 4-5 levels overall each having different classes of bugs, some of more or less subtlety. Fits on one whiteboard / single piece of paper in the end.
There are no syntax errors, just logic. Errors are in categories like: robustness, failure handling (internal or external), data/thread races, unforseen side effects, etc. The more experienced you are, the more problems I expect to be found. A "I don't know" is just fine (and often the following discussion is MORE informative).
In the interview, I'm checking of problems that the applicant notices and see how he approaches the problem.
So far I've been satisfied with all hired who passed that test. And they liked not having to solve stupid whiteboard problems.
Thanks for sharing.
I disagree, somewhat. In positions where I work with a medium or large team, any code of mine you reviewed would give you an idea of how the team leads want code to be written, not really how I as a candidate write code. I've often disagreed on design/architecture but was forced to go through with it because I really didn't feel like arguing and getting on the shitlist of a high-ranking engineer with influence in the company.
If you want an idea of how I would design something, it's probably best to have me write something specific to the job. I'm not talking a whiteboard red-black tree implementation, but something that is actually indicative of the work I'll be doing.
Instead we have to simulate work samples
through whiteboard coding, pair programming
sessions, take home coding projects, coding
knowledge tests, etc.
Very small sample size but we've had good results skipping coding exams entirely, and simply talking through some scenarios with candidates.- give them a hypothetical problem ("the page is loading slowly") and role play how they'd troubleshoot it - do they understand how different bits of the stack fit together? have they ruled out network issues before digging into the code? etc.
- "tell us about a tricky technical challenge you solved"
- ask them what makes a for good code review on a pull request / what makes good code review culture in general
- ask them about things the team did well at their previous role(s) and things they'd improve, and how they'd improve upon them. (not looking for "correct" answers or opinions - but they should have some opinions)
- what are some technologies/frameworks/languages you'd like to learn next? again there's no "right" answer. but if they don't have any - red flag, right?
- etc
This of course presumes that some minimal screening has been done to make sure they're actually a software developer; an initial Fizzbuzz screen etc.
At each step of the way we discussed relevant technical bits in a natural and conversational way. For example if they mention that they "optimized" or "refactored" or "fixed" something at a previous job then naturally we ask them for the details and usually wind up swapping some stories as well.
If they passed this conversational interview, we usually extended an offer to them within 24 hours or less. Sometimes we waited longer and they got snatched up by another company.
Of course this sort of interview doesn't tell you about their work ethic and how reliable they are and so on. But, neither does a lengthy coding assignment-based process either. And the "conversational" method gives you a sense of how they'll fit in with the team.
Considering how quickly good candidates are snapped up, doing "fast" interviews like this can be a big advantage for your hiring efforts. If worst comes to worst, and they don't work out, you can part ways. It's not like either side is locked into the employment agreement. Depends on your specific locale a bit, of course -- and at my current employer we're helped by being a remote team, so it's not like we're asking folks to relocate or anything.
I don't know. I think I'm pretty good at troubleshooting on the job (since I can be the one to troubleshoot & solve the trickier bugs) but thinking about doing it in an interview setting makes me nervous. It feels too fickle & up to the biases of the interviewer that day, not to mention that any serious production bug relies on knowing how the software stack operates & which things are relevant, all things that are ambiguous in an interview the interviewee would just be stuck grasping in the dark to try to figure out which scenario you might be describing.
The rest seems fine, but I'm curious about your experience with the live debugging & if you've considered cutting it.
but thinking about doing it in an interview setting makes
me nervous.
I felt it went pretty well in the interviews and didn't seem to introduce any extra stress.Personally, if I was being interviewed, I'd really prefer this to coding exercises as those really crank up the anxiety for me. (But of course everybody has different preferences!)
not to mention that any serious production
bug relies on knowing how the software stack operates
& which things are relevant,
Oh, I think maybe I didn't communicate this aspect properly in my post.There's no right answer. We're just looking for their thought process and how they might look up and down the stack to zero in on something. Of course they would need to ask questions about the architecture along the way. We do communicate all that up front - it's their questions that we're looking for.
The rest seems fine, but I'm curious about your experience
with the live debugging & if you've considered cutting it.
Well it's only been a handful of developers, but I felt it was helpful.I think that kind of exercise is good for distinguishing "recent code school graduates" who are smart and can write code for one or two parts of the stack, from more experienced devs who have experience understanding how all the pieces of the stack fit together in a production environment.
Just today I discovered the biggest issue we've been encountering on our platform, and I'm not even in a developer role here. The developers didn't track it down since I got them involved, but here I am coming in and figuring it out, and it didn't even require access to the source code or how the platform works.
At the end of the day these types of issues come down to creativity and outside the box thinking than it does knowing how the specific software platform works. Looking at bugs and issues in a way that isn't subject to preconceived ideas and existing knowledge.
Sounds like a fun interview for someone like me lol
1. Do they understand the basic machinery of a web application (assuming the job is about building web applications)?
2. Can they take a vague problem ("the page is loading slow") they might get from an actual non-technical user and drill down to identify what precisely the issue is? Is it DNS? Is it a proxy server/load balancer? Is their latency at the DB layer? How would you tell the difference between all of these?
Especially if you're junior and your current job sucks and you've never (yet) passed an interview for a better job.
This is counter-productive, but the brain doesn't always work how you'd want it to.
If it's not a network issue, it's probably the database, right? Generally you've got an n+1 and/or some individual huge slow queries. I'd interested in knowing how the candidate would zero in on the database possibility and how they might narrow it down from there.
I have so much trouble remembering previous challenges I've overcome. I don't ruminate on them, I just move on...
it is really testing a particular set of social skills
I need my engineers to have strong communication skills, so this is intentional and desired. ...which have minimal bearing on an engineer's ability to do their job
In positions I've held, communicating and remembering are pretty desirable qualities. For engineers and others. I need my engineers to be able to do those things. You obviously don't. That's fine. We'll hire different engineers. Everybody wins.Makes it way easier in interviews or ever just in a normal performance review where you can refer back to things.
I'd be fine if you had to skip a few questions like this and did well on the rest.
That's a fairly common interview question, though, so it's a good one to rehearse!
Namely, if you've had only boring, uninteresting jobs so far, you might not have a truly interesting "tricky technical challenge you solved," and similar gaps around the other questions. Like, you can't name many of the ways a big, high-traffic site might slow down, because the internal tools websites you work on don't need caching or proxies or load balancing or...
Whereas tossing an algorithm question or two at them too to see if they can figure it out at least potentially screens for potential in a way this conversation doesn't.
So I always make sure we do both on interview loops, because I don't want to reject you just for lack of good past jobs, OR because of lack of studying algo problems recently (the good senior candidates crush the conversational one even if they're rusty on the algorithms).
Like, you can't name many of the ways a big,
high-traffic site might slow down, because
the internal tools websites you work on don't
need caching or proxies or load balancing
I feel like it would be tough to spend any amount of time developing web apps, even if they're just internal tools, without hitting a basic performance issue like an n+1 query situation where you accidentally write a page with 500 queries or something. But maybe that's me... maybe devs are better than me these days. I hope so =)There are of course innumerable network related reasons an app could slow down and we wouldn't expect a software dev to dive into them... I'd expect them to just be able to do some ultra basic stuff to rule out a network issue and zero in on the application stack.
As a software engineer for 20+ years, I've talked with my sister, who is an ER doctor about this often. Most folks outside of engineering think our interview processes are crazy.
Keep in mind, there's significantly more training that goes into the medical field, so there's more emphasis on the institutions. Where did you go to school? What hospitals or facilities did you work with? Your resume and references speak more strongly.
Other hurdles in software engineering hiring:
- It's both a technical and (often) a creative field.
- Most work requires teams to complete and there's very little insight into who contributed what into a project.
- You don't have many of the signals from other fields, such as published papers, law cases, or hard numbers such as sales, certifications, etc.
Long story short: we don't have good mechanisms within companies to measure engineering productivity, let alone measure programmers outside of an institution.
I'm reminded of Steve McConnell's intro to "Software Estimation":
"Meanwhile, the typical software organization is not struggling to improve its estimates from +/-10% to +/-5% accuracy. The typical software organization is struggling to avoid estimates that are incorrect by 100% or more."
They do, but not for the reason you think.
A hospital administrator would be mad to hire some rando from the street who calls himself a doctor, who has no formal schooling in medicine, a dropped two-year degree in chemistry, but who reads a few medical blogs, on nothing more than 'They have four years experience working at a startup and a good reference and they can answer some second-year whiteboard questions about the lymphatic system.' [1]
Other professionals front-load the interviewing effort into a rigorous schooling, testing, and accreditation process, that costs hundreds of thousands of dollars and years of your life. Meanwhile, software engineers grouse about whiteboard interviews.
[1] Despite the highly dramatized performance by Leonardo DiCaprio as Frank Abagnale, that's not generally how doctors get hired.
We have certs in software engineering, but in my experience, they've largely failed as an indicator of qualifications. Some hiring managers see them as nice to have, but I've also met hiring managers who are actively skeptical of certifications.
We also don't have differentiation of software levels or skills. If you're a programmer, you could, in theory, go program anything from web to AI to robotics. You don't have to be certified to work in nearly any industry. And we don't have formal hierarchy of skilled work, such as EMTs, nurses and doctors.
We're a relatively young industry that just hasn't worked things out yet and, to be honest, in the United States, there isn't pressure by employers to get our act together. The engineers themselves are in charge often and we're skeptical of boards and authority. We have too many stories of folks making it big with no formal education. Those with the money aren't demanding change because the current system "works."
It's possible software engineering will someday evolve into a boring respectable profession, but right now it's still a creative field of fads, trends, rock stars and super stars (let alone ninjas and wizards).
To me they are a red flag. Someone couldn't get a good job in a bull market straight out of undergrad and felt the need to get additional training...
> We also don't have differentiation of software levels or skills. If you're a programmer, you could, in theory, go program anything from web to AI to robotics. You don't have to be certified to work in nearly any industry. And we don't have formal hierarchy of skilled work, such as EMTs, nurses and doctors.
It still exists, it's just that it's informal.
Hi I'm Ed Barns I'm here for the interview of ER Physician.
QUICK Put on your scrubs we've got a patient here with a bone dangling out of his arm.
No you don't understand I'm just here to interview.
THIS IS THE INTERVIEW HURRY UP.
It's also a big part of residency (having to treat patients with diminishing supervision from senior doctors). So it's not that far fetched.
With software you can write a TODO app with a million lines of code or 7.
That, and the field moving so quickly and being so broad. I interview for ML. I assume interviews for front end are extremely different.
This exists in Canada for some areas of software development that falls under the requirements of being a licensed software engineer.
Many people, including CS graduates in Canada, think it’s ridiculous. The requirement for certification is pretty strict in that you need a computer engineering undergrad degree or a degree in software engineering from an accredited program. This is probably how most people would be accredited, but there are some other ways too. All of them, as far as I can gather, require formal educational credentials.
You then need to go through a formal apprenticeship while working under someone who is licensed. It’s probably an 8 year process in total.
Further to that, there is a limitation on what you can practice. There are different disciplines (e.g., mechanical, civil, electrical) and there are fields and scopes of practice within each of those disciplines. What this results in is someone who is licensed in, for example, embedded systems wouldn’t then be able to go and work on websites or back end databases.
The field is further divided into technologists and technicians that require less education than engineers and have much less autonomy and scope of authority.
Fundamentally though I think that system of high priests and laypeople who are barely allowed to code at all can't be sustained. I strongly suspect all professions will eventually require a little programming, and certification will be hard for those cases.
I'd hire that one over anyone pushing horse de-wormer, anti-malarial drugs, or homeopathic remedies.
Meanwhile in Software there are initiatives for "Gender X of Minority Y can Code" and "Everyone can be an Engineer after 3 months of BootCamp".
> It is much easier for product designers and product managers to pass interviews. You can see their work product.
You don't see any more work of a product manager than of an engineer. What you see is always the finished product, and you have no idea who contributed what parts. If the overall product that can be publicly experienced is good, then some engineers in that team seem to have been doing a good job. You won't easily know which ones - and you especially don't know whether a product manager on the team really a lot of influence on this or whether the products vision and execution was mostly driven by engineers.
And the second point:
> If you could review someone's code from their previous jobs, I think it would be quite easy to vet applicants. And it would be much easier to move from job to job.
Sometimes employers can - because candidates have open source code. But experience has shown that these datapoints are not used, and employers still fall back to the default process of assuming nothing and doing whiteboard/leetcode style interviews.
No one looks at open source work, just as no one cares about your resume.
If you could review someone's code from their previous jobs, I think it would be quite easy to vet applicants. And it would be much easier to move from job to job.
Instead we have to simulate work samples through whiteboard coding, pair programming sessions, take home coding projects, coding knowledge tests, etc. We know that none of these methods are particularly good, so we err on the side of failing candidates when we are on the fence. They also take a long time to complete, so software engineers are reluctant to change jobs because they know how much of a pain it will be to go through interviews again.
It is much easier for product designers and product managers to pass interviews. You can see their work product.
At my company we're working on a way to create standardized coding projects that can't be faked so you can use them across job applications. If you are interested in this space, shoot me a message: robert at facet dot net.