> This actually might not be so nuts if it were clearly a way to evaluate how you handle unfamiliar problems and they made it clear they didn't really expect you to finish the whole thing in 50 minutes, I guess.

This happened to me. I once interviewed at a place where they presented me with a language I had never seen. Very different concepts than your typical C/Algol flavored languages. They went over the basic concepts, with a handy reference sheet for me, then gave me a few tasks to complete in that language.

Towards the end, after completing the tasks, I congratulated them for inventing a ficticious language in order to test candidates on a more equal footing. Cute, clever, creative, even though it's probably far-removed from the real world...

Their response? Oh this is not a ficiticious language, this is what we use daily. It's our proprietary language and we're quite proud of it.

Me: ...

(I ended up getting the job and learned a lot in the domain of wheel reinvention.)

Man, this reminds me of the story about the person that gets a position at a new company and it turns out they have a ridiculous tech stack where it's a custom language. The new person comes in and adds comments which breaks everything and somehow files have like a version history which affects how they run? Anyone have the link to that? There's some individual that developed all of it and gets the new person fired for breaking everything in production (because of course it all runs from head) and was portrayed very negatively.

Yes, thanks :) I thought the TLs name was Tom.

I have a hard time imagining this is real, but it makes some twisted kind of sense.

Was it a company that is in the healthcare technology space, because that sounds like something you'd find there. That or aerospace.

MUMPS?

I find it disconcerting that there is a language the healthcare industry uses named MUMPS.

Very close, yes. And yes to the parent comment. ;)

So it was Perl?

I don't know. I'm very comfortable in JS, valilla, jquery, react and vue. And apparently my customers think I do a decent freelancer work since they keep hiring me back.

And yet, an entire SPA in vanilla JS in 50 minutes seems a lot to me.

I guess it depends of the size of the SPA, but user input + rendering + routing + whatever logic they ask you for the app is a lot of work.

I guess it depends a lot of the requirements: you could skip the routing, make a dirty innerHTML rendering, deal with one browser only for the input events, etc. Still...

Besides, I hate being rushed when I code. I never speed code IRL, the deadlines are in days/months, hours for crisis, never minutes.

I've done a similar test. I didn't finish it in the time allotted, but a few other candidates did and they hired one of them. The fact companies can do these kinds of demanding frontend tests and still hire people is what makes me think there's a glut of web developers and not a shortage like people keep saying.

I had an onsite recently that told me were were going to be building something that had to interact with a postgres database.

I built out the entire backend prior and took copious notes because getting stuck looking up things like “how do I initialize a postgresdb” (something I do maybe once a year, if ever) would eat up time.

I think I did pretty well at that onsite because of the fact that I prepped before hand and they didn’t seem to mind that.

It’s weird to me because in general most engineers are rarely if ever creating an entire stack from scratch and it can be very easy to want to do such a thing methodically and with best practices in mind.

How would I enable hot code reloading? How do I set up PostGres so I’m not just connecting as super user which is obviously a security risk? Etc etc.

To be able to do these tasks under pressure with an engineers mindset is tough and you either have to really drill down and focus and stress that in a real setting you wouldn’t be connecting as superuser but for the sake of time in this particular toy exercise you will be.

Being able to prep was essential, and even so, I still stumbled here and there with unfamiliar syntax.

Interviewing is not a solved problem.

There's a glut of resumes that have something web dev on them, and a scarcity of reliable ways to filter for the candidates who legitimately have the skillset(s) needed for the job.

I've done a lot of these tests and interview projects, passed plenty, didn't pass a few, didn't bother finishing a few, but - weirdly - every place I've been hired didn't use them in the first place.

I've started to view them as a sign of a company you probably don't want to work for.

> I've done a lot of these tests and interview projects, passed plenty, didn't pass a few, didn't bother finishing a few, but - weirdly - every place I've been hired didn't use them in the first place.

Similar here. Strangely, the places that don't do them pay about the same as the ones who do (in the same market—not talking FAANG versus Bob's Printer Service & PC Repair in Madison, WI).

After my latest search my new policy for future searches (at least until the downturn or it otherwise stops being super easy to find jobs) is at least not to do any kind of project or evaluation before a real interview, that is, without a real person from the company taking the same time I am, at the same time. The kind where they send you to some "coding challenge" website or give you some "take-home" project before even talking to you. If they're asking me to burn a bunch of my time to save some of theirs, it means I'm too far down the slush pile and/or they're too bad at interviewing for it to be worth my time.

We use a third party for the initial code screen because we feel like it’s more useful and more respectful to the candidates. We worked with them to pick questions that were representative of the type of “real” coding someone might do at our company, i.e. not algorithms or brain teasers. The company also includes a section assessing how the candidate communicates about bugs they’ve investigated. The screen allows 75 minutes and can be done any time the candidate wants, without someone sitting over their shoulder. When it’s done, the candidate gets the same feedback that we do about their performance. The feedback we get includes the candidate’s solution, a general “score,” and a series of bullets of things the candidate did and did not do well.

Compare this to what we did previously, which was to get one of our engineers on the phone and on a pair coding website to ask brain teasers. The candidate got no followup feedback if they didn’t do well. On top of that, because the people doing the interviews knew that if they said “no hire” it would stop the process in its tracks and the person would never get an on-site, we had something like a 95% pass rate. We evaluated several third parties for this, and felt the one we chose was the most useful and the most respectful of candidates’ time.

I understand that the process isn’t for everyone (no process is), but we’ve gotten generally good feedback from applicants so far about the company we chose (Woven, for the record).

Edit: also just noting that we don’t do any sort of hard cutoff for the scores they send us. We try to evaluate each candidate’s performance in light of their experience, looking at what they were and weren’t able to get done, etc. We had someone recently do amazingly on the first half but not even finish the second: presumably they ran out of time, but we brought them on anyway because we felt like their performance on the first half deserved earned them some further consideration, even though their overall score was quite low.

I figure if my initial contact & a work history that fits very closely with what the posting company's asking for (as it usually does when I personally apply somewhere, rather than letting recruiters bring me stuff) wasn't interesting enough to get someone on the phone and instead gets me directed to a screener robot, they're just Not That Into Me. If/when (next downturn, probably, when the number of applicants vastly outstrips the available jobs) everyone's doing it I'll just have to live with it, but for now, not worth an hour plus for a lottery ticket to maybe talk to someone. Others are excited to talk to me on seeing what I've done and what I might do for them. No need to buy lottery tickets with hours of my time.

Plus, I mean, I'm not as good at coding challenges as I am at actual development work (which is a lot of stuff, some of which is tappy-tappying code into an editor) so if that's the first gate I have to pass before anything else, that lottery ticket I'm buying with my hour typing at a robot ain't likely to pay out anyway, even if I'm actually the best candidate for the job itself, which isn't guaranteed. Luckily there are so many fish in the sea (job offers paying roughly the same) that it's not an issue for now. Hopefully I've fully "leveled up" into titles for which coding challenges aren't the norm in hiring before the next downturn, otherwise I guess I'll be spending some time drilling to preen those peacock feathers, which is what I'd have had to do this time if it weren't such a seller's market for devs.

That all sounds very sensible to me.

You might have seen the same thing, but any title that sounds managerial in some way - even if in the day to day work, it's completely meaningless - goes a long way toward jumping the line and convincing non-technical people involved in the process that you're a good bet.

Some indication that you have the soft skills and social skills to get along well with non-technical staff helps short circuit the whole thing, as well as convince the type of technical staff that put too much emphasis on coding challenge questions, etc., that they need to cool it and consider the whole candidate, not just a score on a quiz.

Yeah, my curse and blessing is that I'm (apparently—I'm just going off feedback and where I've observed myself contributing strongly to projects) really good at parts of software development that don't really come through in coding challenges, even most of the ones that try to be like "real work", and that are also kinda hard to talk about directly without seeming like a bit of a prick. Good taste, good intuition, broad knowledge, a tendency to think in and about whole systems, and (frankly) sufficient general intelligence that connections among all those things just pop into my head. I also seem to be above-average at explaining concepts to others, at writing generally, and other developers on my teams are usually really happy with library interfaces, APIs, and similar things I design & build (see again: good taste).

Meanwhile, during the act of writing code, I crib off existing code to remember how the hell method invocation looks in the current language I'm writing and similar important details. If I haven't touched a language in 3 months I very likely can't FizzBuzz without syntax errors or using the wrong function somewhere or whatever. I put Go on my résumé because I have written quite a bit of it, and can talk to someone fluently about the parts that are likely to seem odd to a newcomer and some of its strengths and weaknesses, but without some cramming ahead of time or a cheatsheet I probably can't demonstrate a bit of that in code. I'll likely mess up keyword order and all kinds of basic things just trying to "hello world".

In short: I've realized, later than I probably should have, I need to get the fuck out of development per se and into something that's still basically developing software since I'm actually pretty good at that, but doesn't judge me on whether I can, in the moment, recall what a print statement looks like in a language I was writing literally yesterday (it's entirely possible I'll blank on stuff like that!). In my defense I've been easily landing jobs doing this stuff since I was 15 so my blinders were, I feel, well justified—it'd just not occurred to me previously that I might shine better, at least in interviews, which is kinda important, by seeking roles a bit adjacent to development rather than in the thick of it, until I got mumble mumble years in the field and started to think about what I'll be doing when I'm, you know, not even sort of young anymore.

[EDIT] and yeah, of course I could put together Anki decks and do daily drills to get better at the parts I'm bad at and it'd certainly help, but another part of this series of mid-career revelations I've had is that if you find yourself wondering why other people are having trouble with something that's easy to you, that's what you should put your effort into, and conversely, if you can find some way to ignore the parts you find harder than most people seem to (not always possible and sometimes you just have to work on things you suck at, sure, talking about ideally) then you should do so, to free up more time for the easy-to-you-but-not-others stuff.

I think the shortage is _good_ web developers, and it sounds like this test doesn't really help identify those.

    > "to evaluate how you handle unfamiliar problems"

The only problem here is figuring out why would anyone work for a company with these practices. Unless I am missing something and it is common practice to build a product without any thinking, research and design.

Almost no companies come out and say "we will ask you some dumb bullshit in the interview and not tell you what we're evaluating or how we're evaluating it" and a huge percentage of the industry interviews like this, at least to some degree. The best I've ever gotten when asking "is there anything I should prep for or expect for the on-site, and how's the interview going to run?" is "bring a laptop, here's a schedule of who you'll talk to and what their titles, oh and consider dressing [some way]".

I think next time I'm just going to ask them if they merge a lot of PRs that look like what they're asking me to write, and if they say yes, thank them for their time and leave.

I think the answer is often 'no' and they've just never thought about it that way. My favorite interview question has a credible business story I can give for why we're doing it this way.

The goal is to see how you approach thinking about it and conducting your initial research. The solution is irrelevant, and falling back to "well we won't do that at work!!" is missing the point.

If you're the type of person who freaks out when we discuss the possibility of using a new tech you're unfamiliar with, I don't want to work with you. There's a certain level of pressure in saying "I know we've never worked with $TECH before, but we're going to use it for $IMPORTANT_PROJECT_WITH_A_DEADLINE" and if 50 minutes of fiddling around with JavaScript is enough to psych you out, you're probably not going to handle the new tech well either when you barely even know what to search for when looking for tutorials.

From my perspective, a big part of the problem is that companies don't let you know they'll be doing something like this (a simple "the 1:00 section of the interview will involve giving you a task using some tech you're not familiar with"), so it's a surprise (interviews are already stressful in ways that real work pretty much never is, "surprises" are deeply unhelpful for evaluating candidates outside very rare and particular kinds of work), and then also often forget to provide extremely useful information like "we know this isn't a familiar language or environment for you—we specifically want to see how you work in something you're not familiar with, so don't stress out if you aren't close to a complete solution by the end, we don't really expect you to be."

So what would you like to be interviewed on? If it's not based on your CV/experience, general CS best practices, neither known nor unknown technologies, nor general problem-solving questions...

People ask all of these for good reasons. You can write literally anything on the CV, so people ask about your experience to check you're not lying (you'd be surprised how many people do lie and at least misrepresent). People ask coding challenges because they expect you to code (and we've all heard horror stories about like the head of IT security not actually knowing how computers work). They ask you general problem-solving questions to test your intelligence, mental models, and behavior under stress/adversity (again, you'd be surprised how many people just suck here, or worse, get insulting etc. ALSO - note that I'm not saying that all such questions are good, they're mostly bad, but with a little bit of thought and planning you can design a good question). People test for basic CS knowledge because it's important in almost all situations (would you hire a brain surgeon that didn't know what a spleen was?).

(Not discounting all the actually terrible interviewing practices, e.g. people asking impossible questions to feel good about how smart they are, etc.)

I don't think I've directly ruled out much in the way of interview questions on in my comments on this thread, I basically just think not telling people the sorts of questions to expect and in which sections of the day, and not making it very clear what you're evaluating and why you're asking the questions, is harmful and inhumane.

A simple "this part of the day will be algo questions, we do expect actual solutions for at least some of the questions, but we're happy to talk through your process with you as you solve them, and none will be more involved/complex/deep-lore than [a couple examples]" would go a long way. For one thing it'd let people just "nope" out of interviews they know they aren't prepped for ("well they should just know they can't do the job in the first place and not apply" that'd work if interview practices/questions and actual work-on-the-job—let alone what's asked for in job postings—mapped even somewhat closely to one another, across the industry, but they very much do not).

I don't think mystery-interviews and having no clue whatsoever what might be covered in the oft-featuring pop-quizzes probably does much for improving the quality of passing candidates, but it does waste a bunch of time (for everyone) and make the whole thing way more stressful.

[EDIT] tangentially related, I find it really weird that most every company says they want people who can learn new things and get shit done more than ones who just know lots of stuff, but act like if they provide enough information that a candidate might conceivably be able to even kind of study or prep for their interview process, that'll ruin it somehow. WTF? If the candidate is not capable of doing much with red-black trees in an off-the-cuff kind of situation because they're rusty or whatever, but then you tell them a week out that you'll be asking some questions about tree and graph structures more complex than simple binary trees, so they brush up over the weekend then ace the interview including some red-black tree stuff which they couldn't have done under those conditions a week earlier, isn't that precisely what you claimed you wanted in a candidate? How is that harmful?

I think I'd like something like this.

The company has a given technology stack and rather than asking for anything which would directly benefits the company asks you to do some task that might be similar to their tech stack, but instead for the public domain / some open source project.

Programming: We've picked 10 problem tickets from various open source projects that are similar to things that might happen at work, show us what you can do in an hour.

DevOps: There's a local charity that needs a small version of infrastructure similar to what we have in house. Here's some old hardware we're donating and the things we'd like to see on it.

etc.

Whenever I've worked with a new $TECH for an $IMPORTANT_PROJECT_WITH_A_DEADLINE the timeframe for getting up to speed with the new tech has been days, not minutes.

I was on a python/javascript team that transitioned to kotlin -- everything went well, better than I would have expected tbh given only 1 of us even had any production java experience, but 50 minutes would not have been enough time to figure out how to usefully set up the IDE, let alone build anything beyond FizzBuzz.

There's a big difference between spending 50 minutes fiddling with that stuff when you're employed and evaluating it as part of the research phase for a project vs. in an interview with a gun to your head.

we only got half of the story here, but if the workplace has a history of switching technology and trying new things often interviewing a specialist was still their mistake, not the specialist's, that wastes everyone time.

> This actually might not be so nuts if it were clearly a way to evaluate how you handle unfamiliar problems

Why not seek applicants who are familiar with the problems they’ll be regularly encountering for the position you’re trying to fill? And as an employee I’d rather interview for positions that will require me to solve problems I’m familiar with.

That’s kinda like the whole point.

Because even if you're familiar with all the tech we use, you're not going to be familiar with the business domain, our cobbled-together solution on top of the tech stack, and the tens or hundreds of thousands of lines of code we nurtured over the past N years.

How you handle unfamiliar stuff is critical when evaluating an engineer. That's like the whole job.

Unless your goal is to churn out copycat CRUD apps and marketing pages your whole life. Then carry on :)

everyone is looking for their flavour of 'full stack' because in the mind of middle managers human resources need to be fluid across organisational projects.