I'd have a hard time implementing my own crypto, but I've learned enough to know how to use it to secure communications, hide or protect information, ensure no alterations have been made to some arbitrary asset, identify an asset's source, etc.

I love working with a well understood and boring RDBMS. It's predictable and it lets you quickly move on to other problems. But you still need to have a good understanding of how it's implemented in order to store and query your data efficiently. If you have a poor understanding of how indexing works, you'll probably have a hard time selecting the right data model.

There's actually lots of fun problems in the frontend world. Try to write a multi-touch gesture responder, it's very tricky to get things right. How about a natural animation system that allows interruptions? CSS animations tend to look unnatural because they're largely time-based, and they don't handle interruptions very well. (Spoiler alert: springs are the magic sauce.)

Learning about compilers unlocks lots of powerful skills too. You can implement your own syntax highlighting, linter, refactoring tools, autocomplete, etc.

Yes. Or, it's actually rather easy - but it's extremely hard to do well. That's why one shouldn't do it (or, you should, but perhaps for toying with and not for production). See, you obviously know enough about the topic to know this! This is exactly one of those nuggets of "broad knowledge" you need as a software developer. How much do you know about crypto:

- You need to know when to hash ansd when to encrypt.

- You need to know the properties perhaps of symmetric/asymmetric crypto and key exchange

- You must know that you never implement any of these algorithms yourself, you only choose from them.

That's about it. You need to know what you don't know (how to write reliable crypto) in this case.

> If you have a poor understanding of how indexing works, you'll probably have a hard time selecting the right data model.

Right. Basically understanding a database is around the amount you need to implement a toy database. You know the difference between an index lookup and a scan and so on. Lacking that understanding means the database is some oracle (haha) you feed SQL and it spits out data. If you do know a bit more you might have a vague idea about how data sits on pages that are oriented into a btree. You might know that how the disk tree is magically updated in both ends to be consistent even if power is switched off mid-write and so on (I don't know, this really isn't my area - I have coded 13 years without a db). You didn't invent or even deeply understand any of these algorithms to the point where you could write them on a whiteboard. But it does help you if someone asks "what happens to users' bookings if I cut the power?" or "will it be faster to join in the DB or later in the language".

Another pet peeve of mine is people who can't identify NP /exponential problems. It happens several times per year that junior colleagues of mine develop solutions that are exponential in time/space, because that's what the problem is.

Them: "Look, I optimized the order in which we pick the X from the list"

Me: "That will take the remaining life of the universe already with 30 items!"

Them: "Dang :( that took me two days to write"

Me: "Do it the dumb way and get off my lawn"

Everyone should write their own crypto library at least once. Nobody should ever use their own crypto library for anything. :)

One person gave me a step by step guide to how he broke it. It was amazing, and incredibly enlightening how hard encryption truly is.

Mathematical analysis of the encrypted data poor encryption easy to break.

This is validation, and in systems engineering (which software engineering both came from and fed back into before we, in software, forgot about it). It's part of the V&V (verification and validation) portion of system development. Verification means ensuring the system is correct with respect to the specifications and requirements. Validation means ensuring that what's being made is actually the correct, desired thing.

We need our software engineers to study systems engineering, where you will find formal methods being applied to the task of developing complex systems, and rather effectively at that.

Very powerful statement there.

I generally resort to Google and then go find the best approach and implement it if necessary.

I taught myself the basics of the underlying stuff (and it helps that I'm an older developer who grew up on Turbo Pascal and C since I do have a working knowledge of what the machine is doing underneath).

Those are rare cases though.

I've read plenty of implementations, but I never "studied" it thoroughly. And that's one of the reason I never went through an interview with a Big 5 company, where you're normally expected to implement i.e. a tree insertion algo on a whiteboard.

In reality if I need a particular data structure I just pull it in from the standard library.

You don't have to be a structural engineer to build a house ;)

But you need to know exactly what parts actually require a structural engineer or structural calculations. You'll be much faster at building your house if you don't have to think about when you call your engineer and when you can just guesstimate. And you are obviously re-using a lot of structural engineer work when you do (because you buy a prefab door where someone already did the calculations on the hinges etc.) Same with not needing to be a computer scientist to do most software engineering, but you need to use a lot of CS work done by others, and it speeds up your work immensely if you know what term to google.

Also, implememting these deep things (trees, linked lists, hash tables, etc.) mean you have a much better understanding of the tradeoffs you do when you use them. Trying to remember some O(N) numbers for various structures is much harder than just spending 1h making a toy linked list, 1h making an array backed list, and 3h making a hash table just ONCE and then you are set for life understanding the complexity of those things.

No, but you're going to need one on call.

I'm reminded of when when I was still living at home and my parents had an extension and garage conversion done.

Two builders did the whole thing, one in his late 40s and one in his 60s, and for the most part everything they did was just grunt work with very little need for craftsmanship. It's just banging together stud timbers, pouring concrete, digging holes, laying and packing store bought materials etc. Sure there's a lot of experience behind doing that safely and efficiently, but it's not rocket science and nothing a confident DIY enthusiast couldn't read-up on as they went along.

However there were 3 times when they had to call in experts. 1) a bricklayer (a surprisingly impressive craft if you don't want your house to look like shit). 2) roofers (you definitely don't want a roof laid by amateurs) and 3) a structural engineer to advise on (and to sign-off on) reinforcing a supporting wall that held up part of the new roof.

Software isn't that different. You need someone who really knows their shit for maybe 20% and the rest you can sort of palm off and can be done by any of ours peers with just a few years experience.

>> and recalls the algorithm to pick when presented with the

>> problem.

In my experience people able to pick the right algo straight away is extremely rare.

Broad knowledge could even identify several different approaches to the same problem, which can be compared on strengths and weaknesses even before trying one.

E.g for a nonlinear an optimization problem you might consider simulated annealing, genetic programming, gradient descent etc. and you need to know which is easy/hard to write, that gradient is good if you have (monotone) gradients etc.

And those didn't really need any requirements apart from basic math.

There is absolutely no reason for a software engineer to learn abstract algebra, infinitesimal math or any of the other dozen courses that you'll never ever use.

And even then, throughout my 10 years now, I can count on one hand the number of times I actually needed to use these things.

For example, a good knowledge of language theory will prevent engineers from creating scripting languages that cannot be parsed. Probabilities are everywhere in machine learning. Many engineers work with by-products of operational research and need to understand the theory behind them to make them efficient. Complex numbers and trigonometry (quaternions etc.) are needed to build even basic 3D engines. Recent probabilistic data structures such as hyperloglog are being integrated into modern database systems. Good understanding of operating systems is useful for security and parallel programming...

You mentioned a few here, I suppose, there could be a full list already somewhere. :)

https://en.wikipedia.org/wiki/List_of_data_structures

https://en.wikipedia.org/wiki/List_of_algorithms

Absolutely. A general understanding of CS is necessary to be a competent software engineer. Just like a general understanding of physics is necessary to be a civil/mechanical engineer.

If there is a VENN diagram, there is definitely an overlap of "theory" and "engineering". But theory != engineering.