> The only reason it seems to even learn them is to pass an interview...

While I agree that the process is broken, a solid understanding of algorithms and data structures is very important for developers at my company. And nobody knows all, or even most of the algos or data structures! But we do have large amounts of data and some pretty complex analytic problems, and a poor design can cost a lot of time and AWS $. Likewise over-engineering something small is a waste of time not just in writing but maintaining it.

I agree that most e-commerce or business web developent probably don't need this level of formal skill.

Whiteboard questions are useful, but not for figuring out trick questions. Instead for figuring out how the candidate approaches problems.

Those look like software design skills. Also the type of systems you describes use something like Hadoop, and use things like Hive and Pig on top of it. Once you get the data out you use Python or R. And you generally use a standard library to work with. That really SQL + general programming.

What new problems are you working on that require you to invent novel algorithmic techniques?

Even with Hadoop, there's still algorithmic decisions that you have to make like whether to choose a stripes or pairs strategy when synchronization is needed[1], or choosing appropriate data structures for real time queries [2]. Sure something like Hive and Pig might work well enough for certain queries, but for some of the more complex queries where writing bare MapReduce or Spark is needed then these data structures and algorithms concerns pop up quite fast.

[1] https://lintool.github.io/bigdata-2018w/slides/didp-part02b....

[2] https://lintool.github.io/bigdata-2018w/slides/didp-part09b....

MapReduce is filled with algorithms. The core system is basically a merge sort engine (the shuffler). In fact, if you ever wondered about the question "sort 2M integers in 1M RAM", well, that was Jeff asking people if they could understand that shuffle was out-of-core.

Another example is lexicographic range sharding, which uses reservoir sampling to compute optimal tablet key split points by doing a constant-space-and-time heuristic sampling over the keyspace.

I used to think MR was just brute force, but it has many levels of algorithms. Probably too many- at some point it because hard to analyze how the system worked because of the various kinds of hedging and recovery strategies.

I noticed that talk comes from a University. I understand thats a totally different domain of work.

Secondly, you are not inventing any algorithm there. You are only using algorithm invented by others.

Thirdly, you are only deciding what solution works better.

Lastly, in an interview you have to invent this algorithm in 45 minutes.

None of this involves you to invent a new algorithm. At least not in 45 minutes. I doubt if the person giving that talk himself did it so quickly.

Does your standard library have persistent functional thread safe radix trees? Because mine didn't so we had to write it.

Even if it didn't, that's really like an exception to most common use cases of those tools.

Which is precisely why testing candidates on such questions doesn't make much sense.

I understand every once in a while something like that needs to get written, but again that's like an exception.

As an employer though, what do you want to happen when your employee runs into an exception? Do you want them to flounder and have no idea how to solve it or do you want them to figure it out by doing some mildly novel design work?

now I'm curious. What's the use case?

Even after choosing a technology (like Hadoop), there's a lot of difference between the best and the worst algorithm to solve a problem.

For example, the reduce phase of MapReduce is pretty much "apply the algorithm you want on this List", and that's where knowing good list algorithms (for example) shine.

Sorry, in almost all cases its really choosing over several ways of Joining the data, not writing SELECT *, not writing cross joins and learning SQL well.

Which why I ask again. Please list your problem which is so novel it requires you to invent a novel algorithm.

Please note statistics is a science that has existed for centuries now. Unless you are in a university, its highly unlikely you have a problem that will need to you to work on something that novel.

I don't understand your point.

It seems to me you mix statisticals approaches and algorithms, meaning you completely ignore the fact that code runs on computers, and that computers have mechanical characteristics, making two implementations of the same statistical approach wildly different in performance.

I used to work at a company where some guys spent literally weeks inventing a fast way to do a dot product over huge vectors. Which doesn't mean they invented the concept of dot product.

>>I used to work at a company where some guys spent literally weeks inventing a fast way to do a dot product over huge vectors.

Very nice that you bought up this point. This is exactly what I was trying to point out. If a team of full time working people took weeks to arrive at a working solution, despite a massive body of open knowledge available as a reference to prior solutions, there is absolutely no way a candidate can give even arrive at a decent direction to the solution in under 45 minutes in a interview situation.

No one is saying algorithms are not useful.

But there is a very big difference between knowing an algorithm and inventing an algorithm. Testing the former is no indication of the latter, as illustrated by your own example.

There is nothing about learning how basic data structures and algorithms work that won't help you in your day to day development.

You might be "fine" without it in a certain category of job, but rather than be dismissive about it you could be excited about having a massive wealth of information that you can use to do your job better.

There will always be something else to learn, wether it be how to apply Domain Driven Design to your work or how to work with graphs when munging data. It's always worth it.

[edit] Run on sentence

> There is nothing about learning how basic data structures and algorithms work that won't help you in your day to day development.

Having a basic understanding of these things is NOT the same as being able to regurgitate them on a whiteboard, under the stress of the interview process.

Exactly, I don't need to know this stuff of the top of my head. I can look it up...

The reason we test for things is to verify you have enough domain knowledge to "look it up". A bad interviewer will ask you to write quick sort on the board, because if we've already gotten to the point of "implement quick sort" yeah we can just look that up. However, if you're given a collection of nodes and asked to do something with them; that's a bit more interesting. If you know nothing about graph traversal, or have never seen a tree (both not uncommon in people claiming to be principle engineers in my industry) how do you even know what to search for? Best case is you're smart and waste time coming up with a new solution from first principles; most likely it just never gets done.

"these are really hard problems that people have studied for years"

Because of this. So that when you encounter a problem, you know how it can be solved efficiently. If you disregard existing knowledge, you are likely to spend a lot of time cooking up something complex and half-assed that creates an unnecessary maintenance load.

Like music is composed of fundamentals, similarly most problems in computing can be decomposed such that familiar approaches can be applied to them. I think the categorization of Aho's "Foundations of Computer Science" is pretty much to the point on which fundamental formulations are practical to know.

>>If you disregard existing knowledge, you are likely to spend a lot of time cooking up something complex and half-assed that creates an unnecessary maintenance load.

Unless you are stuck on an Island coding during a vacation, not one person on earth faces this kind of a situation.

Almost anybody doing anything apart from basic beginner level coding work knows to look up to a solution on the internet if a solution is not scaling already.

"Almost anybody doing anything apart from basic beginner level coding work knows to look up to a solution on the internet if a solution is not scaling already"

This applies to some practical areas, but some are murky and need more than agile google-fu to waid through of an algorithmic jungle to a solution that provides added value to end user.

There is a discipline and domain dependent threshold where the internet suddenly becomes obscure and unhelpful when one is trying to find solutions. In these instances either you are solving the wrong problem, or the problem is quite complex and/or does not have a closed form/ready solution. The best thing to happen in these instances is that one can limit ones input domain just so that it maps to some algorithmic concept or another. It's not rocket science, but in these instances having an intuitive feel toward the big O behaviour and various algorithmic constructs is really, really helpfull.

Sometimes you just need to implement your own fudged QR decomposition and understand a little bit of numerics, or build custom octrees and whatnot. Couldn't really do it without my books. I do CAD and computational geometry stuff, YMMV.

In your area of work, the domain is computational geometry, so asking questions there is perfectly fine.

In fact that is exactly I was trying to imply. Questions need to be about the area of your work, rather than testing candidates on proxy subject totally irrelevant to their jobs.

I'd learn it just for the knowledge, I have no desire to go into the interviewing process again. I agree that interviewing is a broken process.

I would say it depends. I have been working in this industry for 12 years but I don't need to use those knowledge. However, if you read source code of popular frameworks and systems, they use a lot of those algorithms and data structures.

What better way is there to try and get the best assessment of someone’s coding ability in a 45 minute timeframe?

IMO not doing it. Why'd you do it? There are much more important factors than someone's ability to do a whiteboard exercise while under stress. You can e.g. check their Github, ask them to make a simple example project at home, etc.

This. Test them using the tools they will be using every day.

You aren't testing their coding ability though you are testing their ability to remember how to implement a very specific algorithm.