A good answer that strikes a balance between size of data, latency and frequency requirements is a candidate who is able to show that they can choose the right tool that the next person will be comfortable with.
Obviously organically grown Frankenstein programs are a huge liability, I think every reasonable techie agrees on that.
Check out "data science at the command line":
...or put it into SQLite for extra blazing fastness! No kidding.
I had a heck of a time running the server locally before I discovered the CLI.
Lol. Data management is about safety, auditablity, access control, knowledge sharing and who bunch of other stuff. I would've immediately shown you the door as someone who i cannot trust data with.
No need to act smug and superior, especially since nothing about OP's plan here actually precludes having all the nice things you mentioned, or even having them inside $your_favorite_enterprise_environment.
You risk coming across as a person who feels threatened by simple solutions, perhaps someone who wants to spend $500k in vendor subscriptions every year for simple and/or imaginary problems... exactly the type of thing TFA talks about.
But I'll ask the question.. why do you think safety, auditablity, access control, and knowledge sharing are incompatible with CLI tools and a specific choice of file system? What's your preferred alternative? Are you sticking with that alternative regardless of how often the work load runs, how often it changes, and whether the data fits in memory or requires a cluster?
I responded with the same tone that gp responded with. "blows my mind" ( that people can be so stupid) .
If you really must know: I said "blows my mind [that people don't try simpler and proven solutions FIRST]".
I don't know what do you have to gain to come here and pretend to be in my head. Now here's another thing that blows my mind.
Well why don't people do that according to you ?
Its not 'mind blowing' to me because you can never guess what angle interviewer is coming at you. Especially when they use the words like ' data stack'.
Why would you guess in that situation though?
It’s an interview, there’s at least 1 person talking to you — you should talk to them, ask them questions, share your thoughts. If you talking to them is a red flag, then high chances that you wouldn’t want to work there anyway.
As for interviews, sure, they have all sorts of traps. It really depends on the format and the role. Since I already disclaimed that I am not actual data scientist and just a seasoned dev who can make some magic happen without a dedicated data team (if/when the need arises) then I wouldn't even be in a data scientist interview in the first place. ¯\_(ツ)_/¯
"not understanding the scale of "real" big data was a no-go in my eyes when hiring."
> Consulting service: you bring your big data problems to me, I say "your data set fits in RAM", you pay me $10,000 for saving you $500,000.
A lot of industry work really does fall into this category, and it's not controversial to say that going the wrong way on this thing is mind-blowing. More than not being controversial, it's not confrontational, because his comment was essentially re: the industry, whereas your comment is directed at a person.
Drive by sniping where it's obvious you don't even care to debate the tech itself might get you a few "sick burn, bro" back-slaps from certain crowds, or the FUD approach might get traction with some in management, but overall it's not worth it. You don't sound smart or even professional, just nervous and afraid of every approach that you're not already intimately familiar with.
"not understanding the scale of "real" big data was a no-go in my eyes when hiring." , "real winner" ect.
But yea you are right. I shouldn't have directed it at commenter. I was miffed at interviewers who use "tricky questions" and expect people to read their minds and come up with their preconceived solution.
Your entire post reeks of "I'm smarter than you" smugness while at the same time revealing no useful information or approaches. Near as I can tell no one should trust you with any data.
unlike "blows my mind" ?
> As stated the question didn't require any of what you outline here.
Right. OP mentioned it was "tricky question" . What makes it tricky is that all those attributes are implicitly assumed. I wouldn't interview at google and tell them my "stack" is "load it on your laptop". I would never say that in an interview even if I think that's the right "stack" .
You are assuming you know what the OP meant by tricky question. And your assumption contradicts the rest of the OP's post regarding what he considered good answers to the question and why.
I guess it wasn't but even if so, it would be legitimately baffling how people manage to project so much negativity in three words that are slightly tongue-in-cheek casual comment on the state of affairs in an area whose value is not always clear (in my observations, only after you start having 20+ data sources it starts to pay off to have dedicated data team; I've been in teams only 3-4 devs and we still managed to have 15-ish data dashboards for the executives without too much cursing).
An anecdote, surely, but what isn't?
I was never a data scientist, just a guy who helped whenever it was necessary.
No. You qualified it with "blows my mind" . Why would it 'blow your mind' if you don't have any data background.
This negative cherry picking does not do your image any favors.
My comment wasn't directed at parent. I was trying to be smart and write an inverse of original comment. Opposite scenario Where I as an interviewer was looking for a proper 'data stack' and interviewee responded with a bespoke solution.
"not understanding the scale of "real" big data was a no-go in my eyes when hiring."
i was trying to point out that you can never know where the interviewer is coming from. Unless i know interviewer personally i would bias towards playing it safe and go with 'enterpisey stack'
buddy, you're just rolling off buzzwords and lording it over other people
And I am not a specialized data scientist. But with time I am wondering if such a thing even exists... being a good backender / sysadmin and knowing a lot of CLI tools has always seemed to do the job for me just fine (though granted I never actually managed a data lake, so I am likely over-simplifying it).