From the point of view of the interviewee, it's impossible to guess if they expect you to answer "no need for big data" or if they expect you to answer "the company is aiming for exponential growth so disregard the 6TB limit and architect for scalability"

It doesn't matter. The answer should be "It depends, what are the circumstances - do we expect high growth in the future? Is it gonna stay around 6TB? How and by whom will it be used and what for?"

Or, if you guess what the interview is aiming for, state the assumption and go from there "If we assume it's gonna stay at <10TB for the next couple of years or even longer, then..."

Then the interviewer can interrupt and change the assumptions to his needs.

The interview question made it clear it was a maximum of 6TiB. With this, one can assume it’s not growing yet still think the interviewer either doesn’t know this isn’t big data or that they want to test their knowledge.

From what the comment tells, the interview question said that the maximum IS 6TiB. There was no further information given, so I assume it didn't make any assumptions about how it might change.

Even if it would say "it will stay at 6TiB" I would probably prefer a senior candidate to briefly question it, such as "It is surprising that we know it will stay at 6TiB and if this were a real project I'd try to at least sanitycheck that this requirement is really correct, but for now I'll assume this is a given..."

At least if, as the interviewer, I told them to treat the challenge similar to a real request/project and not to not-question the given numbers etc.

FWIW, it's a 2.5 second extra to say "Although you don't need big data, but if you insist, ..." and gimme the hadoop answer.

That's great, but it's really just desiderata about you and your personal situation.

E.g., if a HN'er takes this as advice they're just as likely to be gated by some other interviewer who interprets hedging as a smell.

I believe the posters above are essentially saying: you, the interviewer, can take the 2.5 seconds to ask the follow up, "... and if we're not immediately optimizing for scalability?" Then take that data into account when doing your assessment instead of attempting to optimize based on a single gate.

Edit: clarification

This is the crux of it. Another interviewer would’ve marked “run on a local machine with a big SSD” - as: this fool doesn’t know enough about distributed systems and just runs toy projects on one machine

That is what I think interviewers think when I don’t immediately bring up kubernetes and sqs in an architecture interview

depending on the shop? For some kinds of tasks, jumping to kubernets right away would be a minus during interview.

Recommending kubernetes immediately is an insta-rejection for me, if I was an interviewer

The interview goes both ways.

The few times I've ignored red flags from a company in an interview I've been in a world of pain afterwards.

So give both answers. Itd would happen all the time in college, or in a thesis defense, where one disagrees with the examiner and one finds oneself in a position of "is this guy trying to trick me!"

Give both answers, and explain why the obvious "hard" answer is wrong

Exactly!!! This is an interviewer issue. You should be guiding them. Get the answers for each scenario out of both sets of interviewees and see if they are facile enough to roll with it. Doing a one shot "quiz" is not that revealing....

> E.g., if a HN'er takes this as advice they're just as likely to be gated by some other interviewer who interprets hedging as a smell.

If people in high stakes environments interpret hedging as a smell - run from that company as fast as you can.

Hedging is a natural adult reasoning process. Do you really want to work with someone who doesn't understand that?

Agreed. I don't really understand the mindset of someone who would consider this sort of hedging a smell. A candidate being able to take vague requirements and clearly state different solutions for different scenarios is an excellent candidate. That would be considered a positive signal for myself and pretty much all the interviewers I've worked with.

Sure, but as you said yourself: it's a trick question. How often does the employee have to answer trick questions without having any time to think in the actual job?

As an interviewer, why not asking: "how would you do that in a setup that doesn't have much data and doesn't need to scale, and then how would you do it if it had a ton of data and a big need to scale?". There is no trick here, do you feel you lose information about the interviewee?

Depends on the level you're hiring for. At a certain point, the candidate needs to be able to identify the right tool for the job, including when that tool is not the usual big data tools but a simple script.

Trick questions (although not known as such at the time) are the basis of most of the work we do? XY problem is a thing for a reason, and I cannot count the number of times my teams and I have ratholed on something complex only to realize we were solving for the wrong problem, i.e. A trick question.

As a sibling puts it though, it's a matter of level. Senior/staff and above? Yeah, that's mostly what you do. Lower than that, then you should be able to mostly trust those upper folks to have seen through the trick.

> are the basis of most of the work we do?

I don't know about you, but in my work, I always have more than 3 seconds to find a solution. I can slowly think about the problem, sleep on it, read about it, try stuff, think about it while running, etc. I usually do at least some of those for new problems.

Then of course there is a bunch of stuff that is not challenging and for which I can start coding right away.

In an interview, those trick questions will just show you who already has experience with the problem you mentioned and who doesn't. It doesn't say at all (IMO) how good the interviewee is at tackling challenging problem. The question then is: do you want to hire someone who is good at solving challenging problems, or someone who already knows how to solve the one problem you are hiring them for?

If the interviewer expects you to answer entire design question in 3 seconds, that interview is pretty broken. Those questions should take longish time (minutes to tens of minutes), and should let candidate showcase their thought process.

I meant that the interviewer expects you to start answering after 3 seconds. Of course you can elaborate over (tens of) minutes. But that's very far from actual work, where you have time to think before you start solving a problem.

You may say "yeah but you just have to think out loud, that's what the interviewer wants". But again that's not how I work. If the interviewer wants to see me design a system, they should watch me read documentation for hours, then think about it while running, and read again, draw a quick thing, etc.

Once had a coworker write a long proposal to rewrite some big old application from Python to Go. I threw in a single comment: why don't we use the existing code as a separate executable?

Turns out he was laid off and my suggestion was used.

(Okay, I'm being silly, the layoff was a coincidence)

because the interview is supposed to ask same questions as real job, and in real job there are rarely big hints like you are describing.

On the other hand, "hey I have 6TiB data, please prepare to analyze it, feel free to ask any questions for clarification but I may not know the answers" is much more representative of a real-life task.

Being able to ask qualifying questions like that, or presenting options with different caveats clearly spelled out, is part of the job description IMO, at least for senior roles.

I once killed the deployment of a big data team in a large bank when I laid out in excruciating details exactly what they'd have to deal with during an interview.

Last I heard theyd promoted one unix guy on the inside to baby sit a bunch of chron jobs on the biggest server they could find.

Is this like interviewing for a chef position for a fancy restaurant and when asked how to perfectly cook a steak, you preface it with “well you can either go to McDonald’s and get a burger, or…”

It may not be reasonable to suggest that in a role that traditionally uses big data tools

I’m not sure if you are referencing it intentionally or not, but some chefs (Gordon Ramsey for one) will ask an interviewee to make some scrambled eggs; something not super niche or specialized but enough to see what their technique is.

It is a sort of “interview hack” example that’s been used to emphasize the idea of a simple unspecialized skill-test that went around a while ago. I guess upcoming chefs probably practice egg scrambling nowadays, ruining the value of the test. But maybe they could ask to make a bit of steak now.

This is sort of the chef equivalent of fizzbuzz or "reverse a binary tree" - there's no gimmicks, something "everyone" should know how to do, it's nothing fancy, just the basics of "can you iterate over data structures and write for loops competently" - or in this case "can you not under/over cook the eggs and can you deliver them in the style you say you're going to.

The egg omelet is a classic for testing French chefs. It is much less about the ingredients (quality, etc), and much more about pure skill to cook it perfectly.

What would be the equivalent for a technical interview? Perhaps: Implement a generic linked list or dynamic array.

I see it more like "it's 11pm and a family member suddenly wants to eat a steak at home, what would you do?"

The person who says "I'm going drive back to the restaurant and take my professional equipment home to cook the steak" is probably offering the wrong answer.

I'm obviously not a professional cook, but presumably the ability to improvise with whatever tools you currently have is a desirable skill.

Hmm I would say that the equivalent to your 11pm question is more something like "your sister wants to backup her holiday pictures on the cloud, how do you design it?". The person who says "I ask her 10 millions to build a data center" is probably offering the wrong answer :-).

Idk, in this instance I feel pretty strongly that cloud, and solutions with unecessary overhead, are the fast food. The article proposes not eating it all the time.

I think more like, how would you prepare and cook the best five course gala dinner for only $10. That requires true skill.

The fancy cluster is probably slower for most tasks than one big machine storing everything in RAM. It's not like a fast food burger.

It's almost a law "all technical discussions devolve into interview mind games", this industry has a serious interview/hiring problem.

You shouldn’t guess what they expect, you should say what you think is right, and why. Do you want to work at a company where you would fail an interview due to making a correct technical assessment? And even if the guess is right, as an interviewer I would be more impressed by an applicant that will give justified reasons for a different answer than what I expected.

>Do you want to work at a company where you would fail an interview due to making a correct technical assessment?

How much do they pay? How long has it been since my last proper meal? How long until my rent is due?

Its great if the interviewer actually takes time to sort out the questions you have, cause seemingly simple questions to you have a lot of assumptions you made.

I had an interview "design an app store". I tried asking, ok an app store has a ton of components, which part of the app store are you asking exactly? The response I got was "Have you ever used an app store? Design an app store". Umm ok.

Red flag - walk away. Interview did it's job.

Engineering for scalability here is the single server solution that you throw away later when scale is needed. The price is so small (in this case) for the simple solution that you should basically always start with it.

Architecting for scalability doesn't mean disregarding resource costs, though - rather, the opposite. In fact, resource costs are even more important at higher scale, because more resources cost more.

I would have had that uncertainty that you are describing when I was a junior dev.

But now as a senior, I have the same questions and answers regardless if I’m being interviewed or not.

You don't need to guess, you can just ask the interviewer. (Shocker, I know.)

But the interview is for a data science position. Why play games with "Ha ha 6 TiB can fit on my PC we actually don't need you?"

> just that they got tricked in this very artificial situation where you are in a dominant position and ask trick questions

DING DING DING!

About 30% of all interviews I've had in my career the person doing the technical interview was using it as a means to stroke their ego. I got the impression they don't get the power they want there and this was a brief reprieve where they are "the expert" and get to thumbs up or thumbs down in their kingly intellectual facade.

TBF, I personally cut them some slack because they may actually be right about a lot and not given the authority to do what they know is right.

Still, if we're subjected to AI resume filtering, then how about we replace the technical interview process with AI too and eliminate the BS ego trips?

I want my boss to be straight with, I need those below me to call me out of Im talking bullshit.

Tell me this isn't big data, and then, if you must, tell me about hadoop (or whatever big data is)