They claimed it would take only a couple of hours but for the 15 or so bullet point requirements that was simply not realistic. Spent a couple of days on it and the only feedback I got was no thanks.
They had previously stated they pay for every stage regardless of outcome. After sending my details they never paid me.
Feels vastly hypocritical given their stated people first work culture.
I really meant the "you might enjoy" part. I wasn't suggesting it's an efficient way to make money.
Some people are really bothered by misdeads going unpunished.
I even took the time to learn enough Perl to write the solution. They said it wasn't required but would be a nice touch. It was an interesting project to work on and it was fun coming up with a solution. I thought it fulfilled their requirements pretty well, but also just got a short "no thanks" email.
I was left wondering for a long time what they didn't like about it..
Someone suggested they may use these interview submissions as cheap labor to get ideas to solve internal problems. I thought that was ridiculous until I reread the disclosure I had to sign at the beginning:
> [DDG] will be the exclusive owner of all right, title, and interest in and to the work product resulting from the project, including all intellectual property and proprietary rights.
Could have just been boilerplate disclosure stuff, but seems a little weird.
Don't get me wrong if you had fun (and I enjoy them too) nothing wrong with that.
I also use them as starting points for the next take home assignment and even live interviews (I’ve been able to keep the project open in the IDE on a separate monitor and copy and paste code even when doing a camera-on + screenshare interview)
Here's another: https://github.com/fly-hiring/platform-challenge
Was it my issue waiting a short time to start it? Maybe, but you’d think they’d at least try to answer a question or two. Oh well, I just gave up because if they weren’t going to put in any time, why should I?
If they said it would take 4 hours and you spent 10 hours on it, it’s likely your approach was ineffective compared to a more skilled engineer.
The point of the assignment isn't to make it "fair" (in some loose sense), but to filter out candidates.
Let's say I come up with an assignment and I pass it to a junior to see how long it takes. The junior does the work, likely reports the time rounded down, but also with no stakes. Jim job is secure.
As an interviewee you're gonna spend time polishing the code. Every variable well named, lots of comments, good functions, well considered parameter order etc. It's gonna take longer because , for you, the stakes are higher. You'll likely build some code to test it, and so on.
This is before we discuss domain knowledge that the junior had, and also that the company implemented the task in the first place and didn't just spitball the time expected.
They promised this person if they completed it, they would get an actual human response back.
Not only did they not get a response back, they didn't get a response back from anybody, even when directly trying to follow-up.
And this was what was promised for spending 4 hours of your time on something.
At the very least, that seems unfair.
The most fun task I had so far, the feedback was pretty good as well. Kudos to tptacek for that.
Maybe they had bad experience with sharing feedback directly?
Correction: It's not profitable.
The less feedback they give the less embarrassing it is likely to be.
At the core of everything people-related a company does or does not do is the fear of a lawsuit, and in the case of interviewees, the only thing that leads to a lawsuit is telling the interviewee anything at all - therefore, no feedback.
"We didnt hire her because she seemed like she might get pregnant" isnt the kind of feedback most people write anyway and if they did a 2 minute pass via HR is enough to eliminate the risk.
The companies that are actually worried take pains to tell you what not to say during the interview. This is rarer than you'd think. Most companies aren't actually that worried about getting sued over interviews. That doesnt stop them from tossing this lame excuse at you though.
And of course, this is more likely when the interviewee submits a more advanced or higher-level solution, meaning the best strategy is to "dumb-down" the submission rather than trying to use the latest language features.
---
Unfortunately, we’re not going to be moving forward with your application for the backend role at this time.
The first thing we want you to know is that we got a lot of applications for these roles. Like, a lot a lot.
The way we evaluate candidate submissions is that we’ve built, over a year or so of running this challenge, a written rubric for what the strongest submissions look like. We started with something sane looking, and then iterated over time as we got a sense of what candidate submissions actually looked like.
We score submissions according to that rubric. Different members of our platform team score different applications; they’re just jumping in and looking at the code and making grading decisions. In the pool of candidates we’re evaluating right now, your code submission missed our cutoff.
A natural thing to want from us next is a copy of the rubric, or specific information about what your submission was missing. We’d want that too! But we can’t give that to you, for (at least) two reasons:
(1) The outcome on your code was a weighted combination of a bunch of different factors, so there isn’t a simple answer that doesn’t just dump our whole rubric to you.
(2) We want to preserve the opportunity for people to apply to these roles in the future, and spoiling the challenge would break that.
What we can tell you is that the most successful submissions had a combination of these attributes:
- Heavy user focus. Successful submissions explained how the system could do good things for users. - Straightforward database models that make the queries we care about fast to run. - Transactionally safe sync with Stripe. For example, the best submissions mentioned that Stripe API requests had to be idempotent to ensure the user is not charged multiple times.
We sincerely hope you stay in touch and re-apply in the future. In the meantime, we'd like to give you some Fly.io credits to play around with! If your email address here matches your Fly.io email, click here to get them. If not, please let us know which email you use for Fly.io and we'll set that right up.
Please feel free to reach back out to us about other roles, or about this role in the future. Thank you, again, for doing this.
> When you’re ready, let us know and we’ll schedule it for review. We review submissions once a week. You’ll hear back from us no matter what by the end of the /following/ week, possibly sooner if you submit early in the week.
+ 1 sentence expressing "we will tell you if you passed or failed, we have a policy of not providing actionable feedback"
> Remember, we are not timing you and there is no deadline. Work at your own pace. Reach out if you're stuck or have any questions. We want you to succeed and are here to help!
+ 1 sentence "but make sure to not spend significantly more time than X hours"
The best way to reject a candidate IMO is to call the candidate, deliver short feedback over the phone “Some of the feedback we got from the interviewers was we like to see the problem solved in a shorter amount of time/ less space complexity/ whatever). If I was on the fence, I would them know that if they want to try again they can give it another shot in 6 months, a year, whatever. To me this seems like the most humane way to do it (This was actually my experience with Google, maybe 8-10 years ago).
The job listing promised mountains and that it will take few hours and they’ll take time in their feedback,etc.
except i just got a generic reply that “we’re not satisfied with the solution”. For something that took ~10 hours, i was at the least expecting a vague pointers on whys. I even sent an one pager explaining style decisions, caveats, etc. It was pretty insulting.
And when asked for a follow up, got a more generic bs about how the evaluation criteria is honed from their years of experience and that is not something they share outside.
One good thing that is it made me realise interviewing still sucks and i just stopped looking for jobs.
These should not take 10 hours, unless you're learning Rails from scratch or something (some people do this). We've tried different ways of saying "don't spend more than 2 hours on this unless you really want to" and it doesn't always come through.
Kind of appalling how callous you are about wasting candidates time because they are not effective mind readers. Did you ever worry that you would become one of those mildly sociopathic executives that don't treat your fellow human beings with respect the moment it doesn't personally benefit you?
I don't know what fly's internal processes are, but I suspect the reviewer had to have submitted some bit of internal feedback on each submissions on why they're okaying it or not. A filtered version of it or even a separate candidate specific feedback would be just enough.
I've been at the hiring side of the table for many years and every time we share a handcrafted feedback to interviewees that we have passed on, we always heard back good things. They know we respected their time (and them), even if they don't agree with the particular feedback.
This feedback is not meant for them to get "better" or something. My point is, it could even be 2 hours the candidate spends, the least they deserve is knowing that it was actually looked at.
The submissions that we continue with check a bunch of boxes. The submissions that we pass on don't.
The problem is, we're not hiring people to build to a spec. We want to assess your decisions and ability to go from a basic problem to a first implementation. If we shared the rubric with folks, they'd focus entirely on trying to check the boxes and I don't think we'd get an accurate assessment.
Your point about valuing peoples' time is important, though. We have not yet found the right balance for everyone.
I wonder if your hiring process is skewed by the perfectionists who spend tens of hours of their submissions. Arguably that's the type of thing you guys would want to select for but it's not at all fair for those who timebox themselves to 2 hours or whatever.
I don't think you can spend more hours and get these criteria.
It's pretty rare IME to encounter interviews which are closely related to the actual work you'll be doing, mostly it seems like the main purpose is to filter out the heaps of bad/mediocre developers.
This sounds like a classic "my developer reckons he can do this in 2 hours (but never actually has)". The reality is your developer is crap at estimating.
A good developer will be lucky to produce 100 lines a day. if you doubt that check your git history (excluding autogenerated crap).
So unless the project you're setting people needs, on average, 25 lines, whatever you're setting people is clearly not a 2 hour project. And that's not even taking into account whatever hurdles you've inadvertently put in the task, strange build config, obscure libraries, out-of-date libraries, non-standard formatting, etc.
Could you tell us what the average length of a submisson is?
One task I got given a year or two back had an old version of Vue, linting rules that conflicted with the defaults of the 'usual' IDE you'd use, wanted you to create 3 new backend API endpoints and a new frontend page with non-trivial functionality. Plus they wanted units tests.
That's a 2 day job. They also claimed it would take an hour or two.
You could have told the candidate that, then. That would have been some useful information to have for the hours the candidate put into their submission. Or like "hey, we're going to just skim over this with a subjective list of check boxes".
I have enough experience that I probably would have smelled this before I got that deep into the process and noped out, but for the hopeful mid-level/new-senior devs this sounds pretty demoralizing.
If you ever get asked to do one of these exercises, it’s useful to try to determine where they are in this process. Ask about their hiring timeline, how long they have been interviewing for the position, and try to get a feel for how fatigued they are in general.
If it feels like they are just starting, ask how long you have to complete the exercise. If there isn’t a time limit, it’s better to wait as long as possible, so they can reject other candidates first and burn themselves out. If there is a time limit, ask to pause the interview process (insert some excuse here) before they send out the exercise.
If they are near the end and sound exhausted, that’s a good sign and the effort might not be wasted.
What detail is shared about what "checkboxes" the candidate met and didn't meet? From the parent post and your reply, it doesn't sound like much at all?
I've had a few bad experiences when sharing feedback with candidates myself and I would understand doing the checkbox approach for feedback and/or just never sending detailed feedback, but actually grading submissions pass/fail based on a subset of criteria you jealously guard from candidates essentially selects for lucky people. If I wanted to do that, I'd just shuffle the submissions by number of bytes and discard everything that's a multiple of 5 or something.
Did you try paying candidates for the home assessment? People's time costs money. Paying people not only helps attract candidates but also helps the company reduce the list of candidates to take the assessment.
I'm inclined to say the interview process is imbalanced with more power in the hands of the employer. If you're looking for the right balance, try turning over some power from the employer to the candidate.
The candidate's cost was 2-hours worth of time. Was the employer's cost 2-hours worth of time?
I'll make sure we get you a response within the next week.
I doubt anyone will read frustrated comments and my responses and think "oh boy, I didn't want to apply there but that one guy responded and made me change my mind".
Isn't that ironic, indeed hypocritical? You take feedback from job applicants, consider it super valuable, but refuse to give anything useful to those very job applicants.
Feedback was something like "Assignment is private and you can't store code on GitHub, you should send zip archive"
As far as I remember, I was told it would take a few hours, maybe 4, but the assignment looked rather fun, so I thought, whatever, let’s do it.
Maybe I didn't understand the assignment fully or missed some cues; I don't know. But it took me roughly 10 hours spread across two days. After 5-6 hours, you don't feel like throwing the work away; you just want to finish. It was pretty frustrating.
After returning the assignment, I waited the two weeks they asked for and heard nothing. I sent one email, waited a week, got no reply, sent another, waited some more, and still received no reply. I ended up sending a handful of emails to various addresses I could find. I even sent a DM on Twitter to one of the founders to let them know. No reply anywhere.
Overall, it was a pretty bad experience.