So much focus has been put on 10x this and high performing that. Companies waste lots of engineering talent and company money looking for 10x when 10x is mostly situational in nature. Set the scene. In one scene the engineers are not producing. Change the scene and now they are 10x.
Instead of looking for 10x, that time would be better spent making incoming and current engineers better at their jobs and creating 10x situations.
I can relate to this. I have pretty good resume - good schools, advanced degree, impressive sounding projects, a long list of publications. My track-record suggests I am at least a 3x engineer.
So when I get into a coding interview, I'm expected to perform as such... but I usually come off as a 0.3x engineer. Like really bad. Forget syntax bad. Forget algorithms bad. I've bombed so bad in a technical interview, the interviewer questioned if I even knew basic trigonometry. They had spent 10 hours interviewing me prior and felt good, but got me in person and suddenly thought I was an idiot.
In each of my jobs, I've had a considerable ramp-up period. I'm talking 6 months or more of feeling completely lost and unproductive. I get in my head and it makes me perform worse.
If I'm not fired in that first period, I'm lucky. Only after about a year on the job I can perform at the level my resume would suggest.
I regularly look things like that up. I am a builder and a problem solver, not a reference manual.
"I'm your Google, anything you'd need to look up just ask me and I'll give you the answer".
Great way to take the pressure off plus it gives you a good idea on how well they ask questions/are comfortable leaning on others. It's a huge plus if I can get a reasonable candidate to admit that there might be parts of the problem at hand they don't understand.
I've written code tests where I'm expected to read the code, find the issues and fix them on paper. But when they take it too far beyond the spirit of what I believe to be a reasonable test of understanding and fixing code, then I write that I'd just for example compile the program and look at the output.
The companies that are most into questions like these seem to be the classic (failing agile) body shops with ridiculous turnover of their talent, which causes them to have to interview even more, which causes in turn for their tests to become even more abstract and worthless.
6 months to get into things is kinda long though. Unless it's super complex things, it should be less. Are you sure you are not exaggerating?
Most engineers can be productive in a month. The 3x engineers take half that.
Do you see how that putting-words-in-others-mouths looks? It's rude and does not contribute too any kind of productive discussion.
I respect your opinion but I find this strategy very strange. (Or maybe I don't exactly understand what you're saying.)
In my view, it would be a dream scenario if the next 10 programmers I hire were all superior to me. If I was the worst programmer on the team, that would be an ideal outcome. Yes, I've been programming for 30+ years but I'm also self-aware of my limitations. This gives me the ability to recognize better programmers and learn from them. If I'm not hiring interns, I do expect (or at least hope) that they are better than me.
Am I competent enough to write a "do/while" loop and knock out FizzBuzz in 5 minutes?!? Sure, but that doesn't mean there aren't better programmers than me. I would consider it a great business skill if I'm able to consistently attract superior programmers -- either to work with me or for me.
Larry Page attracted programmers better than him (e.g. Jeff Dean). Yes, Larry earned a masters degree in computer science but his java programming (the rudimentary 1996 crawler) wasn't considered very good. His employees rewrote the whole thing in C++ and made it scale for billions of pages. Bill Gates hired superior programmers. Mark Zuckerberg hired superior programmers. Etc. etc.
It doesn't seem like it. The next 3 sentences from the thegayngler:
> I have but one qualification. Can they do the job? Are they strong enough that I can guide them into the position I need them at if it is required.
In my mind, programmers superior to me have abilities that would be beyond my knowledge to "guide" them. They can come up with programming solutions I could never think of. It's very realistic that "the qualification to do the job" essentially means they are a better programmer than me.
Presumably, in your mind, "programming" skill is along one spectrum and one focus, as well? I think that's part of what's being talked about here. There are many, many aspects to software engineering/programming (depending on whether you want to classify them differently), and many roads that could have been taken to get to the point someone is at. I see no reason to think that someone that is better or worse than me in one aspect will also be so in every one of the other aspects of the job.
Let me try to restate that in more general terms and you can tell me if I interpreted it correctly: Since programming skill is multi-dimensional / multi-faceted, and it is unlikely for a programmer to be superior in _all_ dimensions compared to another, it is therefore not possible to conclude if one programmer is superior to another.
Is that a fair rewording?
My response is... sure, programming skill can be looked at as multi-dimensional. But so can other real-world comparisons. We prioritize the dimensions that are more important and collapse all aspects into a composite score.
Multidimensions such as choosing a new job: Company X is a 5-minute commute with 4 weeks vacation but Company Y has the newest technology and has lucrative stock options. Yes, it already goes without saying that there's more than 1 aspect of "desirable working conditions" at each company and no single company can claim a superior score on _all_ aspects. We can still weigh all aspects and eventually pick one (and only one) which is superior.
If I hired a "Jeff Dean" type of programmer, it would be an upgrade to my team and my company. Could I search for an "aspect" that I'm better at than him? Well, looking over his expertise[1], it looks like he doesn't have much frontend GUI programming experience. So yes, I guess I could claim superior skill on writing event handlers in C# language for buttons. That's not important though for my assessment. My point is I don't need for programmers to be superior to me in every aspect for me to determine that they are superior overall. (Same as determining which company is "superior" to work for.)
If I was founding a startup and I was the "ceiling" of programming skill instead of the minimum "baseline", my company would fail. I think I'm pretty decent at programming but I have no Dunning-Kruger delusions that I can "train" anybody that can pass FizzBuzz into a "Jeff Dean". I know of no 6-week bootcamp or even a 6-month corporate training program that can claim that feat. What I can do is recognize the programming skills in others that are (overall) superior to mine and hopefully convince the candidates to come work with me.
[1] https://scholar.google.com/citations?user=NMS69lQAAAAJ&hl=en
Well... no. But I think that's my fault, for the most part. I read the part of your original comment that said 'In my mind, programmers superior to me have abilities that would be beyond my knowledge to "guide" them.' as implying all their abilities as being beyond you, but it doesn't specifically say that.
I wasn't trying to imply that it's foolish to try to ascertain if one programmer is generally better than another (as long as some common domains along which to measure are involved), just to note that it's possible to hire someone that's far beyond you in some areas, and behind you in others, with the intention of specifically helping them build in the areas they are lacking (which have application to the job). The result should be a programmer that is, generally, as good or better than you in most areas, making them a more obviously superior programmer, even if that wasn't entirely obvious at the time of hiring.
In short, you can hire for the things that are important that you can't easily improve while allowing deficiencies in the areas that are less important which you feel you can help them improve in.
As a strategy, it has it's pros and cons. As a pro, it's probably easier to find candidates since your criteria is less stringent. As a con, you might find the candidate you hired has problems coming up to speed in those areas they are deficient in (which may be why they were deficient in the first place, instead of lack of experience).
I didn't really express the point very well originally, and just stopped at the opening, probing question.
Ive seen companies literally turn good engineers into underperforming engineers...stifling them with process, tech debt, politics, etc... Then they get angry some people can perform and others can’t under the circumstances and conditions the management created.
I’ve watched perfectly good engineers get fired or forced out for not being miracle workers only for people to later realize there was nothing wrong with the engineer in the first place.
You don’t know who is going to be the best until you work with them. I would advise people to pick people they will like even under the worst of conditions and train them to the way you want them to be.
I also find it remarkable how quick people are to blame the engineer, rather than considering the whole picture (process, tech debt, politics, etc). It's a complicated equation with a large number of variables.
In my experience the manager cares about fit and attracting you to the team, whereas the peers are all looking for reasons NOT to hire you.
Jokes aside; I agree with the advice of hiring people smarter than you in the domains where you won't be paying attention should the company actually take off.
If you're looking to be a founder/CEO level type, doesn't seem to matter what your skill in Java and OOP are, yes? No need to be a water walker.
That's the "genius" of management; they sell an idea and get someone else to do the work.
Imagine a programming ecosystem with e.g. 5 approximately equally popular frameworks. Suppose that someone with this attitude is familiar with frameworks F1 and F2, but never used F3, F4, F5. Put such person in charge of an interview, where equally skilled candidates (i.e. knowing 2 of the 5 frameworks) apply. They will only classify 10% of them as "senior-level" (those knowing F1+F2), and additional 60% as "somewhere between junior and senior level" (those knowing one of F1,F2; and one of F3,F4,F5). So they would hire literally themselves, but probably not someone on the same level.
IIRC a Google recruiter once sent a set of packets to a hiring committee and all of them got a "no hire". Then the recruiter revealed the packets were lightly-anonymized versions of the actual packets of the hiring committee members, from when they'd all originally interviewed.
I wonder how much of that comes from being exposed to all of the consequences any poor decisions they made shortly after they were first hired.
This is so correct.
Productivity follows when people are working for a worthy cause. In fact the core of productivity isn't lists, management techniques or whatever, it is a thing called 'constancy of purpose'
There exist no such a thing called 10x burger flipper.
If you don't encourage a culture where everyone is asked to hire people better than themselves, it can easily degrade where each progressive hire is slightly worse.
I was shocked doing an interview in my last round and got so many test questions. I bombed so hard because it wasn't what I was doing.
The real key to me is, we treat all of these jobs as more or less the same, when in reality they are all vastly different. There is no singular senior engineer skill set, but a variety.
I wrote a whole blog post on that: http://www.mooreds.com/wordpress/archives/2812
I just wish companies would be more upfront about what they really needed, as opposed to the wishlist.
I was declined for a position at a trading shop for a C# backend role because I used linq to do something. I didn't have unlimited time for the tests, so I went with expedient over performant, for which they did not care. Fuck that, set clear expectations for interview "tests".
If your priorities aren't aligned with the environment you set up, you are the one that must say so. On a real world project that means once in a while I'll take a project back for small fixes, instead of wasting unending hours thinking about every possible problem up-front... if you expect something different, well, that's not optimal.
This really seems out of necessity. There are going to be a lot more qualified median engineers rather than superstar coders, so it makes sense to optimize your hiring pipeline to assess these folks.
In my experience, most of the "top 10%" hires are either through interpersonal contacts or a fast-tracked interview process for a specific candidate.
At least when middle managers do it they're only interested in who gets credit for the work so they leave the average tech guys alone long enough to get the work done.
In that case, the key was that clumps of them had already worked together in earlier jobs, so the incompatibilities had been pruned.
If you got an interview, and did well you got another challenging programming problem to solve on-site. You were given a computer with the programming environment of your choice and as much time as you wanted, as well as someone to ask questions from.
I think it was also important that the company solved hard graph problems in Lisp, with a lot of macro-driven code. If you were a Lisp programmer, you generally wanted to work there.
You can pretty easily hire the top 1% of applicants, and still get only mediocre results. E.g. at one job, when looking for a senior-level eng position, clearly stated in the description, we easily weeded out a couple hundred applicants who had effectively zero experience whatsoever. No prior programming-related jobs, internships, side projects, open source, nothing.
Which is a waste of time for everyone involved. Hence the heavy reliance on connections :(
In reality, the applicant pool is heavily biased towards very bad engineers.
Exp: You must code in an unfamiliar language (Fortran 1988, for example) and do something simple in it. At least then you'll know that people are all starting from scratch. Time it, make sure it doesn't last more than 2 hours, see how laughably slow we all are, pick those that get the furthest.
Even being upfront about our low expectations and providing very friendly and helpful people to pair and guide, it lead to record numbers flipping the table and stomping off premises before the completion of the interview.
I don't use this approach now, as I don't think its a good use of time (and I think an interview should be a positive experience even for those that fail), but it did provide, I believe, a fairly strong signal of the attitude with which a candidate approached learning new things (but not really anything else).
Really, please, tell us more!
It's not that you're wrong, it absolutely is a really good way of finding out how someone reacts to being asked to learn something new in a stressful situation. And that is a valuable thing to know - I love working with people who get excited about an opportunity to learn something.
It's just that you have to be prepared to give so much help and the question has to be so easy because it's all completely new that you can't test competence in any meaningful way. Which means that at the end of the 90 minutes or whatever, you know exactly one thing more - their attitude to learning under pressure, but you have relatively little time with them, and you probably need to learn multiple things per session.
You also have to be careful when in the interview you throw this. A bad hit like this can throw a candidate off for the rest of the interview even if they are quite acceptable.
The Kobayashi Maru test.
1) Some people may find Fortran easier to understand than others even if they haven't touched it before. This was the case when my University decided to use Haskell* as the first language to teach students in order to level the playing field. It did help somewhat, it leveled the field for those with little and those without any prior programming knowledge. But to those who had already come to terms with recursion etc it was a very tiny stumbling block.
2) Some people need to dive deeper and gain a greater understanding of things before they feel at ease using them, but once they have reached that level, they will often outperform most other people.
* This was 20+ yrs ago and Haskell was not so well known back then.
nice to see it's more common
Oh wow, that would definitely get me sweating. All the FORTRAN i learned was 77!
* Predictive of job performance
* Either (1) difficult to optimize against or (2) easy to optimize against but in a way that doesn't reduce predictive value
* Low cost to give
* Low cost to take
How about "Google is giving tests in these 25 languages"? They could even choose languages that are purposefully dissimilar from any language that you've claimed familiarity with.
After that he said "Okay, I know you know enough to handle the job, now I'm curious how much you really know. " (This is key, I feel. He specified I wasn't going to get dinged for not knowing the answers, so I didn't feel the pressure in the questions he asked afterwards, when I usually feel intense pressure, as I know I've been passed over for getting a single question wrong in multiple interviews in the past.) He then asked me some pretty deep questions about low level memory and other things, and if I said I didn't know he turned into teacher mode and taught me the concept.
I actually walked out of that interview having learned something new and useful. I then got to chat with the President very casually, and I received a job offer a few days later.
The interviewer new his stuff, too. He'd been programming arcade games in assembly for decades. His games have sold millions of dollars, and two of the series still get made today.
Ever since, I figured if a freaking legend could be satisfied after reviewing some code and a code sample, the crap I've had to endure everywhere else is completely and totally unnecessary, and it's just frustrated me to no end.
After several questions that he was sure I knew the regular stuffs and was qualified for the job, he said he was going to ask some really hard questions to see how much I really knew and what was my thought process. The questions had no simple answers. Each of them required knowledge and experiences from a few technical fields. I didn't know much then. He guided me how to solve the problems, discussed the pros and cons of each approach. Then he told me how they did it, how other products in the same category did it, etc.
The half-hour interview was prolonged to one and a half. At the end, he said, "You have a very intuition on what are the right directions to go when facing unknown complex issues", and gave me an offer. It is my best interview experience. (I didn't join the start-up due to other reasons.)
The technologies have been advanced in a blazing fast speed i the last 10 plus years. Technical interviews, on the contrary, regress in a similar speed.
One company I interviewed with gave me a program that had to be written in Scala, with the full knowledge that I would have to learn Scala to do it. Please.
I actually really like homework assignment approach when done correctly, both as an interviewer and as an interviewee (though seanmcdirmid's suggestion of prep material, rather than a homework assignment, is quite sound), but this is a separate issue, independent of how skills are assessed.
It's really about the company/team having a preconceived notion of what skills they are looking for vs. looking to see if a candidate has a distinctive skillset that could help round-out the team. How do you, as the candidate, market yourself and the things you are good at?
Obviously, the resume is a good place to do such marketing (and I'm sure you already are) so if teams are turning down applications based on the homework without looking at the other skills the candidate has to offer, then that might be an issue.
"The IT industry is committed to increasing the number of women engineers. But if they're too busy caring for their children to spend six hours doing our silly test, that's their fault for being women. We are forced to hire 20-something white males who have nothing better to do with their time."
For her at least, this worked out better for her. She does design, which is even more difficult to evaluate in 30 minute time slots.
Not that I’m arguing this is the way to go, I was just wondering if interviewers could give some hints on what they would cover I the interview, so we could prep like studying for a test.
I hope your not impugning the motives of those who issue take-home assignments. Existing practices for hiring are objectively bad at selecting qualified candidates, and most people are just trying to do better. There is some evidence that take-home assignments are an improvement above the norm.
Also, consider that such assignments can serve as a "blind-audition" and can help eliminate bias from the interviewing process. And there are lots of ways in which loading everything into the interview unfairly penalizes certain types of otherwise qualified candidates. What's left, referrals? That's definitely biased, even if it also tends to have a lower false positive rate.
An easy switch would be to convert take homes to onsite work. One of the more recent interviews I had consisted of putting me in a room with some data and a stub of code and asked me to fill in the stub over a few hours. Every so often, they’d come by and ask if there were any problems. (More frequently at first, and then later less frequently.)
It’s basically the same, but it’s timeboxed and well defined with enough back and forth, to not feel like they’re just throwing a problem over the wall to you.
They are relevant enough that the prior poster felt the need to smear them.
> if the effect is the same
The claimed effect is, at this point, purely hypothetical and indeed runs contrary to my experience.
> An easy switch would be to convert take homes to onsite work.
Generally speaking, an onsite task like you describe seems like a reasonable choice for a company to make. However, this strikes me as worse for the affected groups that reaperducer was concerned about:
- It requires scheduling an even larger fixed block of time outside one's normal job/life responsibilities. A take-home assignment at least allows for flexibility for when you work on it. It strikes me that time-poor candidates would appreciate the flexibility.
- It would no longer be a "blind audition". When I've reviewed take-home assignments, I don't know anything about the candidate except that the code. It strikes me as a good way to resist unconscious bias from seeping in.
The the on-site solution remains a blind audition if the people doing the evaluation are not the people administering the test. This is exactly the case for a take home assignment, only now it is timeboxed and with access relatively timely help if needed.
The purpose of the blind audition is not to hide the identity of person being interviewed, as much as it is to enforce an objective standard and remove bias. If the rubric is, “Accept all that got automatic evaluation score of at least X”, then that achieves a similar goal.
As far as time-poor people, never underestimate the desire of having a clearly defined block of time. Of course, onsite or not can always be optional as well.
Pointing out (potential) unintended consequences is fine, even welcome.
Putting sexist remarks in the mouths of others is a smear, even if it is done merely flippantly.
> While it is true that the effect is merely claimed, as I know of no relevant study, your anecdote equally meaningless.
Anecdote is more meaningful than mere speculation. Even without statistically valid data, examples that are contrary to our expectations can bring to light factors that we had not previously considered.
> The the on-site solution remains a blind audition if the people doing the evaluation are not the people administering the test...The purpose of the blind audition is not to hide the identity of person being interviewed, as much as it is to enforce an objective standard and remove bias. If the rubric is, “Accept all that got automatic evaluation score of at least X”, then that achieves a similar goal.
A fair point.
> This is exactly the case for a take home assignment, only now it is timeboxed and with access relatively timely help if needed.
The point about timely help is definitely a point in favor of on-site work tests. But working in a timebox can also be stressful. A take-home assignment without artificial deadlines (i.e. don't schedule any interviews until after the assignment was received and approved) can eliminate that stress.
> As far as time-poor people, never underestimate the desire of having a clearly defined block of time. Of course, onsite or not can always be optional as well.
I think this hits on an important point, and one that I think would benefit from research and expertise I don't have. I'd hazard a guess that it depends upon why one is time poor.
If the candidate is intelligent and capable, but somewhat disorderly, then they would probably benefit from a structured, on-site task.
If the candidate is more organized but simply has a difficult and rigid schedule (which is actually something I would expect true, for example, of a working mother) then a take-home assignment might be better.
I don't know that either one is superior in all cases. Honestly, I think the best option would be to give the candidate a choice.
You will never be unbiased in a hiring process, because the bar you set is not universal, and always be biased in some way. Being "as blind as possible" isn't the right metric, and we shouldn't hire based on it.
Yes, that's exactly what I'm doing.
I wrote a thing about it and I got someone who responded to me with a rather scathing rebuttal. I don't know, it's very conflicting.
Post in case you're interested: https://dev.to/spirodonfl/should-i-accept-coding-challenges-...
Edit: better wording
However, don't see how well take home can work for senior engineers. In fact, its unfair and delays interview process to expect take home assignments from working folks.
A valuable practice, which surprisingly few tech companies do, is to ask candidates about diverse and orthogonal problem domains, or alternatively, allow the candidate to pick from a diverse set of problem domains. In the former case, you often find that candidates have difficulty writing even simple solutions for some problems but on others they instantly and fluidly can code up a good solution. Because they've had to do it in recent memory and still have the "muscle memory" from doing it previously. You often see bipolar results across the problem set this way.
For senior engineers with deep domain expertise in an area, there is an additional trap in that they use more sophisticated and often very different algorithms in their day to day lives than are applicable to the toy problem domains. Graph algorithms are a good example of this, and they are popular in interviews. The representations most engineers know (e.g. adjacency list or matrix) don't scale but the extremely high scale algorithms operate on a different set of principles that aren't trivial to code and aren't relevant to non-parallel cases; coding up an adjacency list graph traversal algorithm is going to be very unnatural to a software engineer used to doing the same on trillions of edges in real-time. It would be crazy not to hire an engineer on this basis but I've seen it happen, ironically because their expertise caused them to show poorly on the coding exercise.
Interviewing for technical skills is intrinsically difficult but I think that as an industry we are much worse at it than we could be. For all the claims of companies that they only hire the "top 10%" or whatever of engineers, the interview process is often optimized to the benefit of the median engineer.