News for hackers. Stuff that matters.
Re:How to hire low experience, high potential people by tptacek on Wednesday February 07, 2024 @12:03PM (#39291022)
We built Fly.io resume-blind and without interviews, hiring people at every level of experience without having to make decisions based on that experience. We did it by throwing away all this stuff, ditching interviews, and replacing them with work-sample tests. Some of the best people on our team, the best I've worked with in my whole career, are working here in their first job in our field.

I'm a little grumpy about this "diamonds in the rough" shit. I'm more concerned about what's lurking in the diamond mines. If people can demonstrate that they can do the work, I don't much need to know if "they have a chip on their shoulders". More generally: I have zero faith in anyone's ability to learn much from psychological interviews.

Re:How to hire low experience, high potential people by swozey on Wednesday February 07, 2024 @12:42PM (#39291570)
So you communicated via email and the only consideration was the quality of their sample?

Nobody at any point literally spoke to this person?

We've had a ton of issues at my corp of people having other people handle their interview questions, code, or having AI listen to the conversation and answer questions by text as if we prompted it. Then they show up online after being hired and they have webcam problems or never go on cam, talk as little as possible, basically never communicate to anyone really, miss a lot of meetings, etc.

We've had probably 5 instances of this last year we're absolutely super strict about things now. Our assumption is that they want to get in, potentially get any bonuses if there are any, get a few huge (for their call center) paychecks until they get caught. Do no or almost no work. And it's a call center you can sometimes hear tons of convos going on in the background so they're just doing this en masse.

This is happening ALL over since covid. I think some of them may have even filed unjust termination sort of things to the workforce commission but I may be wrong. Huge companies are dealing with this at a really rough scale.

Re:How to hire low experience, high potential people by tptacek on Wednesday February 07, 2024 @12:45PM (#39291604)
Our process runs mostly through email until the last challenge, which is interactive (for every role in the company we simulate some kind of design or problem solving exercise on Slack with an random member of our team, which is then rated based on a rubric by a panel that reviews the transcript; it differs from an interview in a variety of ways, most notably that the team member delivering the exercise doesn't ever interrogate the candidate).

Most of our work sample stuff is design/architecture based. I'm not worried about how LLMs impact it.

This is the sole way we've hired engineers at Fly.io since 2020 and we have never once brought someone on who turned out to be 3 raccoons in a trenchcoat. Raising the bar further on ourselves: we hire globally, at SF scale salary worldwide.

Re:How to hire low experience, high potential people by potamic on Wednesday February 07, 2024 @02:32PM (#39293124)
That's an interesting approach. In an interview, you generally drive the discussion a certain way to gather data points for your rubric. Wouldn't this happen in your case? If yes, how is it different from an interview? Otherwise, how do you evaluate candidates who solve the problem but don't seem to communicate much about it.
Re:How to hire low experience, high potential people by tptacek on Wednesday February 07, 2024 @02:38PM (#39293196)
Nope! It's complicated. We put a lot of effort into deprogramming candidates before the simulation (we call it a "workday"). If you join the Slack channel for a workday and wait for us to ask you questions, almost nothing will happen. The candidate drives, apart from the brief we provide at the beginning of the session.

The rubric heavily draws from what questions the candidate asks, and which topics they choose to zoom in and spend time on. It's interesting; I warn candidates that studying for it ahead of time (for instance: for a long time our workdays were all based on a block storage problem) puts you at a disadvantage, by robbing you of good questions to ask. It's not a big disadvantage, but interview-style preparation isn't doing much to help you either.

I'm always reluctant to talk too much about the workday process, in particular because the more I describe it the less chill it sounds. In reality, it's you joining a Slack channel with an engineer from our team and just shooting the shit about something we're (in the simulation) planning to build.

A lot of what that challenge is evaluating is communication. We're an all-remote company with team members in virtually ever time zone; being able to work out in the open in the environment we've created is important.

I think our workdays are a pretty quirky feature of our process. You could instead do a scripted interview, where everybody gets precisely the same questions or prompts. The big thing we do with the workday that I think everyone should clone is: deliver the interview over Slack, and don't allow the person delivering the interview to grade it; instead, have someone else grade from the transcript. It was like a light went off (on?) for us when we figured out to stop having workday participants offer performance feedback about candidates.

(One thing you immediately get from it is a process to iterate on the quality of your staff delivering the workday/interview; you can immediately see which team members click with candidates and which don't, and then nudge people in the right direction. It is wild to me that people do free-form interviews without this kind of iteration system in place.)

Re:How to hire low experience, high potential people by afpx on Thursday February 08, 2024 @03:31PM (#39307279)
I love that high-profile people like you are taking risks and innovating with hiring. Thank you. I'm hoping to see some change in the industry. There are probably a lot of high-performing people who have little motivation to go through the normal interview process.
Re:How to hire low experience, high potential people by tptacek on Thursday February 08, 2024 @06:30PM (#39309277)
In fairness, there are people who show really well in the "conventional" interview-driven process that aren't super motivated to go through ours. We come close to achieving a free lunch, but still only approach it asymptotically.
Re:How to hire low experience, high potential people by tasubotadas on Wednesday February 07, 2024 @05:32PM (#39295214)
Could you give some exercise examples? How complex do you make? Does it feel like doing work for you?
Re:How to hire low experience, high potential people by tptacek on Wednesday February 07, 2024 @06:14PM (#39295615)
It feels like what working here feels like, but it is strictly not doing work "for" us.
Re:How to hire low experience, high potential people by tasubotadas on Thursday February 08, 2024 @07:06AM (#39300993)
How different are the take home exercises from the workday?
Re:How to hire low experience, high potential people by mplanchard on Wednesday February 07, 2024 @12:47PM (#39291627)
Do y’all have any blog posts on this? Would be curious to know more about the work sample tests, and about the process in general.
Re:How to hire low experience, high potential people by tptacek on Wednesday February 07, 2024 @12:47PM (#39291640)
Re:How to hire low experience, high potential people by strich on Wednesday February 07, 2024 @12:34PM (#39291460)
In a world where chat gpt exists I think even work samples are untrustworthy sources of excellence unfortunately. It would seem one has to either watch the process or devise unique and difficult to game tests.
Re:How to hire low experience, high potential people by tptacek on Wednesday February 07, 2024 @12:35PM (#39291477)
Work samples aren't all straight coding; in fact, only a small subset of ours are.

Later

I want to add: I fully believe LLMs can reliably knock out the code for most programming challenges, but a good coding work sample is as much about what you choose to code as it is about the code itself.

If you're asking people to invert binaries trees or whatever, that's not a work-sample test, it's a coding quiz. Work-sample tests mirror the actual work you're doing; that's what makes them predictive. Writing a function to do X, Y, or Z is usually the easy part about building software; the hard part is deciding which X, which Y, and which Z, and how they fit together.

We didn't deliberately set out to design work samples where the decisions we're grading are things LLMs don't just do for you; we do a lot of systems programming work and those kinds of decisions just dominate systems programming.

Re:How to hire low experience, high potential people by xena on Wednesday February 07, 2024 @01:41PM (#39292434)
As a fly.io employee, I can tell. I can tell fairly quickly.
Re:How to hire low experience, high potential people by BadHumans on Wednesday February 07, 2024 @12:37PM (#39291501)
Is this how Fly still operates? If you're hiring Rust developers, are you open to someone who has never touched Rust in their life?
Re:How to hire low experience, high potential people by tptacek on Wednesday February 07, 2024 @12:42PM (#39291571)
We wouldn't know if you've never touched it, only whether you can solve problems in it today.
Re:How to hire low experience, high potential people by mcguire on Wednesday February 07, 2024 @03:01PM (#39293501)
So, no?
Re:How to hire low experience, high potential people by tptacek on Wednesday February 07, 2024 @03:17PM (#39293691)
I don't think this is a question with a yes/no answer? If you apply for a role that involves writing Rust and can't demonstrate the ability to solve problems in Rust, you won't make it far. But we don't even collect resumes, so you get to make that call for yourself; we don't divine it from signs and sigils.
Re:How to hire low experience, high potential people by dentemple on Wednesday February 07, 2024 @08:29PM (#39296718)
I wish more companies hired software engineers like you.
Re:How to hire low experience, high potential people by randomdata on Wednesday February 07, 2024 @05:08PM (#39294936)
Implying that those who have never touched Rust cannot solve problems?
Re:How to hire low experience, high potential people by mcguire on Thursday February 08, 2024 @02:45PM (#39306656)
Implying that those who have never touched Rust cannot solve problems in Rust.
Re:How to hire low experience, high potential people by randomdata on Thursday February 08, 2024 @03:48PM (#39307484)
Do you consider Rust to be the problem solver's kryptonite? If not, it is not clear why a problem solver cannot solve the problem of solving problems in Rust.
Re:How to hire low experience, high potential people by BadHumans on Wednesday February 07, 2024 @09:56PM (#39297397)
I guess I misunderstood what you are getting at. In my mind, hiring someone who has never used X language but can, in general, solve problems and program so we'll hire them and they'll learn X language was what I thought you were talking about.
Re:How to hire low experience, high potential people by jefc1111 on Wednesday February 07, 2024 @01:24PM (#39292193)
I don't know if I would hire like this, but I do think a lot about the current hiring landscape is pretty broken, so maybe I should! I expect an employer needs a certain amount of cachet before being able to do this or anything else similarly left-field.

Side note - I started down two paths of significant new learning early last year: Security & Elixir / BEAM. I love Security Cryptography Whatever and had no idea you crossed over into the Elixir world (fly.io). Cool!

Re:How to hire low experience, high potential people by tptacek on Wednesday February 07, 2024 @01:26PM (#39292224)
I've been at Fly.io since 2020. I wouldn't describe us as an Elixir company so much as a company that appreciates Elixir. Some of our backend is in Elixir but the majority of it is Rust and Go.
Re:How to hire low experience, high potential people by jefc1111 on Wednesday February 07, 2024 @01:33PM (#39292326)
Some significant names from that world on the fly.io roster :) Like a lot of people I feel like Elixir / BEAM has a real value proposition that is unrealised in the wider IT landscape ... but maybe that's not such a bad thing from some perspectives.
Re:How to hire low experience, high potential people by TylerE on Thursday February 08, 2024 @03:08AM (#39299519)
It’s a really a shame that Java poisoned the GCed JIT Runtime well.

Now everyone assumes that a runtime means it will take 30 seconds and 8GB of RAM to show a loading screen.

Re:How to hire low experience, high potential people by geor9e on Wednesday February 07, 2024 @01:43PM (#39292458)
Plus you get a bunch of free work out of every applicant if you delegate real issues out as work-sample tests :P I'm pretty sure the new Audi A8 cup holders are from my mechanical design engineer third round volkswagen interview a couple years ago
Re:How to hire low experience, high potential people by tptacek on Wednesday February 07, 2024 @02:02PM (#39292740)
You do not in fact get free work.
Re:How to hire low experience, high potential people by gwbas1c on Wednesday February 07, 2024 @05:53PM (#39295434)
It's nearly impossible to scope useful, shippable work that can be accomplished in a job interview. Otherwise, you're asking for 20+ hours of work, which means that only fools / suckers will do your interview.

Do you really want a company full of fools and suckers?

Re:How to hire low experience, high potential people by geor9e on Wednesday February 07, 2024 @07:08PM (#39296046)
Shippable, goodness no, but a 1-hour-limit prototype already solving a problem you haven't even begun thinking about? I'd take that. I tend to start over from scratch after the first prototype anyway. You're interviewing people anyway, your choice is either real issues or fake issues, and fake issues get your nothing.
Re:How to hire low experience, high potential people by i_h8_lc_grind on Wednesday February 07, 2024 @02:06PM (#39292784)
This is good to know. I think a lot of people are tired of wasting their lifespan on grinding lc-medium/hards.

Is there a website with a list of companies that don't participate in the leetcode/FAANG interview process? I feel like there is a need for one.

Re:How to hire low experience, high potential people by em-bee on Wednesday February 07, 2024 @04:15PM (#39294413)
i seem to remember an HN post about such a site, or at least one about a site that evaluates companies hiring processes. also related, about companies development practices.

try searching the HN post history. (and report back if you find something :-)

Re:How to hire low experience, high potential people by swifthesitation on Thursday February 08, 2024 @04:55AM (#39300178)
Re:How to hire low experience, high potential people by i_h8_lc_grind on Friday February 09, 2024 @02:29PM (#39319304)
Beautiful. Thank you.
Re:How to hire low experience, high potential people by gwbas1c on Wednesday February 07, 2024 @05:54PM (#39295445)
Just remember that all interview techniques have bias; what works well for one company / role may not work well for your company / role.

That being said, it sounds like your interview technique works well for you.

Re:How to hire low experience, high potential people by tptacek on Wednesday February 07, 2024 @06:04PM (#39295544)
That may be, but I don't believe interviews work well for anybody.
Re:How to hire low experience, high potential people by firstplacelast on Thursday February 08, 2024 @08:05AM (#39301475)
I started an interview at fly like 1.5 years ago.

Figured I’d give it a shot, they were very responsive, gave me the challenge and I spent more time than was suggested. They had a few core technologies I was not familiar with at all, but figured I could hack it…I could not.

Anyway, I really appreciated it. Walked away with nothing but positive feelings.

I have spent enough time applying/interviewing over the last decade that I am often in a dilemma when I get an offer that I don’t want bc the process is so fucky/I already have negative feelings. From HR setting expectations and then not meeting them, the laid-out interview process not lining up, remote/hybrid changing, inability to describe actual projects I would work on, not meeting people I would work under/with, made to feel like I lack intelligence by interviewers, etc.

I’m not a “rockstar” programmer and never have tons of offers, but I have turned down offers because of screwy interview processes/interviewers just to stay at the evil I know.

Re:How to hire low experience, high potential people by gwbas1c on Saturday February 10, 2024 @10:10PM (#39332354)
Don't feel bad about saying "no" to a job you don't want. I've walked away from a few. Sometimes it's money, and sometimes it's just a bad smell.

I did get a decent offer once, declined, and later learned I dodged a bullet. Another time I was impatient and took an offer against my better judgement, and it turned out poorly.