Even with Hadoop, there's still algorithmic decisions that you have to make like whether to choose a stripes or pairs strategy when synchronization is needed[1], or choosing appropriate data structures for real time queries [2]. Sure something like Hive and Pig might work well enough for certain queries, but for some of the more complex queries where writing bare MapReduce or Spark is needed then these data structures and algorithms concerns pop up quite fast.

[1] https://lintool.github.io/bigdata-2018w/slides/didp-part02b....

[2] https://lintool.github.io/bigdata-2018w/slides/didp-part09b....

MapReduce is filled with algorithms. The core system is basically a merge sort engine (the shuffler). In fact, if you ever wondered about the question "sort 2M integers in 1M RAM", well, that was Jeff asking people if they could understand that shuffle was out-of-core.

Another example is lexicographic range sharding, which uses reservoir sampling to compute optimal tablet key split points by doing a constant-space-and-time heuristic sampling over the keyspace.

I used to think MR was just brute force, but it has many levels of algorithms. Probably too many- at some point it because hard to analyze how the system worked because of the various kinds of hedging and recovery strategies.

I noticed that talk comes from a University. I understand thats a totally different domain of work.

Secondly, you are not inventing any algorithm there. You are only using algorithm invented by others.

Thirdly, you are only deciding what solution works better.

Lastly, in an interview you have to invent this algorithm in 45 minutes.

None of this involves you to invent a new algorithm. At least not in 45 minutes. I doubt if the person giving that talk himself did it so quickly.

Does your standard library have persistent functional thread safe radix trees? Because mine didn't so we had to write it.

Even if it didn't, that's really like an exception to most common use cases of those tools.

Which is precisely why testing candidates on such questions doesn't make much sense.

I understand every once in a while something like that needs to get written, but again that's like an exception.

As an employer though, what do you want to happen when your employee runs into an exception? Do you want them to flounder and have no idea how to solve it or do you want them to figure it out by doing some mildly novel design work?

now I'm curious. What's the use case?

Even after choosing a technology (like Hadoop), there's a lot of difference between the best and the worst algorithm to solve a problem.

For example, the reduce phase of MapReduce is pretty much "apply the algorithm you want on this List", and that's where knowing good list algorithms (for example) shine.

Sorry, in almost all cases its really choosing over several ways of Joining the data, not writing SELECT *, not writing cross joins and learning SQL well.

Which why I ask again. Please list your problem which is so novel it requires you to invent a novel algorithm.

Please note statistics is a science that has existed for centuries now. Unless you are in a university, its highly unlikely you have a problem that will need to you to work on something that novel.

I don't understand your point.

It seems to me you mix statisticals approaches and algorithms, meaning you completely ignore the fact that code runs on computers, and that computers have mechanical characteristics, making two implementations of the same statistical approach wildly different in performance.

I used to work at a company where some guys spent literally weeks inventing a fast way to do a dot product over huge vectors. Which doesn't mean they invented the concept of dot product.

>>I used to work at a company where some guys spent literally weeks inventing a fast way to do a dot product over huge vectors.

Very nice that you bought up this point. This is exactly what I was trying to point out. If a team of full time working people took weeks to arrive at a working solution, despite a massive body of open knowledge available as a reference to prior solutions, there is absolutely no way a candidate can give even arrive at a decent direction to the solution in under 45 minutes in a interview situation.

No one is saying algorithms are not useful.

But there is a very big difference between knowing an algorithm and inventing an algorithm. Testing the former is no indication of the latter, as illustrated by your own example.