Re:Stop requiring specific technology experience for senior-...
by JAlexoid on Monday November 08, 2021 @03:47PM
(#29154024)
Well... Funny enough. The person that our team of Spark engineers is cleaning up after was - a C++ dev. For some reason we, Spark engineers, had to rewrite his all Spark application into different components with relevant tech.
Thanks for generalizing Spark engineers to an insulting level, though.
Re:Stop requiring specific technology experience for senior-...
by hippich on Monday November 08, 2021 @10:06PM
(#29157245)
I can't know for sure, but I think person you are replying to meant that in a team lacking diversity (in this particular case, diversity of expertise) all solutions tends to receive same, less than optimal treatment no matter what. If anything, it is insult of managers who create such bubbles, not engineers stuck in these bubbles.
Re:Stop requiring specific technology experience for senior-...
by morelisp on Tuesday November 09, 2021 @02:32AM
(#29158813)
> Thanks for generalizing Spark engineers to an insulting level, though.
I have no clue how you got that from what I wrote. I even said a Spark project needs a Spark dev!
It really sounds to me like you're the one generalizing from "the one C++ dev we hired couldn't do data engineering right at all" to "senior programmers are dangerous to us unless they are Spark developers" (which imo, is more indicative of serious deficiencies in your architecture / design process), to "all senior programmers crossing subdomains are dangerous."
An example close to my heart. The company I'm currently at had a totally ossified Spark monolith and data eng team because they only wanted Spark developers. Drop one C++ developer in there to start spreading the good news about data-oriented design and suddenly there are a dozen previously-intractable "big data" problems we can rip out of the ball of mud and run outside of the cluster. Drop in another SRE who doesn't know shit about Java but can actually write shell scripts, and hey alerts from bad orchestration from weekly to twice yearly.
Yeah if you have a Spark job, you probably need a Spark dev. But if you have a team of Spark devs, suddenly everything that involves more than 1MB of input data looks like a Spark problem and you're in for a real bad time in a couple years.
C++ programmers know map/reduce/filter too.