who, today, can write or optimize assembly by hand? How about understand the OS internals? How about write a compiler? How about write a library for their fav language? How about actually troubleshoot a misbehaving *nix process?

All of these were table stakes at some point in time. The key is not to understand all layers perfectly. The key is to know when to stop adding layers.

Regarding your points, I actually would expect a non-junior developer to be able to write a libary in their main language and understand the basics of OS internals (to the point of debugging and profilling, which would include troubleshooting *nix processes). I don't expect them to know assembly or C, or be able to write a compiler (although I did get this for a take-home test just last week).

Being able to glue frameworks together to build systems is actually not a negative. If you're a startup, you want people to leverage what's already available.

I like to get deep into low level stuff, but my employer doesn't care if I understand how a system call works or whether we can save x % of y by spending z time on performance profiling that requires good knowledge of Linux debugging and profiling tools. It's quicker, cheaper and more efficient to buy more hardware or scale up in public cloud and let me use my time to work on another project that will result in shipping a product or a service quicker and have direct impact on the business.

My experience with the (startup) business world is that you need to be first to ship a feature or you lose. If you want to do something then you should use the tools that will allow you to get there as fast as possible. And to achieve that it makes sense to use technologies that other companies utilise because it's easy to find support online and easy to find qualified people that can get the job done quickly.

It's a dog-eat-dog world and startups in particular have the pressure to deliver and deliver fast since they can't burn investor money indefinitely; so they pay a lot more than large and established businesses to attract talent. Those companies that develop bespoke solutions and build upon them have a hard time attracting talent because people are afraid they won't be able to change jobs easily and these companies are not willing to pay as much money.

Whether you know how a boot process works or how to optimise your ELK stack to squeeze out every single atom of resource is irrelevant. What's required is to know the tools to complete a job quickly. That creates a divide in the tech world where on one side you have high-salaried people who know how to use these tools but don't really understand what goes on in the background and people who know the nitty-gritty and get paid half as much working at some XYZ company that's been trading since the 90s and is still the same size.

My point is that understanding how something works underneath is extremely valuable and rewarding but isn't required to be good at something else. Nobody knows how Android works but that doesn't stop you from creating an app that you will generate revenue and earn you a living. Isn't the point of constant development of automation tools to make our jobs easier?

EDIT: typo

Lets say you have a network performance issue because the framework you were using was misusing epoll, set some funky options with setsockopt, or turned on Nagle's algorithm. A person can figure it out, but its gonna be a slog whereas if they had experience working with the lowest level tools the person could have an intuition about how to debug the issue.

An engineer doesn't have to write everything with the lowest level primitives all the time, but if they have NEVER done it than IMO that's an issue.

The point being that maybe it’s fine if there are a lot of people who only know how to glue frameworks together if they know enough to build useful products. Let all of them try; some of them might very well make it.

1. Working as a programmer perspective: I worked at a company with good practices but so-so revenue. What happens: horribly underpaid salary, nice laptop (but not the one I want), nice working conditions. I am now working at a company with pretty great revenue and mediocre practices. What happens: good salary, I get the laptop I want (not the one I need), working conditions are mediocre.

2. UX perspective (I did a bootcamp for fun): UX'ers make throwaway prototypes all the time in order to validate a certain hypothesis. When that's done, they create the real thing (or make another bigger throwaway prototype).

I feel this is the best approach, from a business standpoint. This also means you have different kind of developers and it depends on the stage what kind they are. I'd separate it as prototype stage, mid-stage and massive scale stage.

This is what most software development is becoming. We are no longer building software, we are gluing/integrating prebuild software components or using services.

You no longer solve fundamental problems unless you have a very special use case or for fun. You mostly have to figure out how to solve higher level problems using off-the-shelf components. It's both good and bad if you ask me (depends at what part of the glass you're looking at).

Thankfully I can use these cool processors to build the next CandyCrush and shine in our modern and innovative society.

Some argue it's unnecessary complexity, but I don't think that's correct. Even individuals want more than a basic geo cities website. Businesses want uptime, security, flashy, etc... in order to stand out.

That's what I expect from someone who graduated from a serious CS/Engineering program.

I'm going to parrot the GP: "That's what I expect from someone who graduated from a serious CS/Engineering program."

I know there are a lot of really bad CS programs in the US, but some experience implementing OS components in a System course so that they can "hack into the OS when needed" is exactly what I would expect out of a graduate from a good CS program.

That's surprising. Recent grads?

The discussion centers on the following expectation of graduates from strong CS programs.

> having working knowledge and being able to hack into the OS when needed.

Now, the course from the listed schools may prepare some students, but I am simply reporting that I have met numerous graduates who state very explicitly.

- they are not comfortable with a variety of operating system concepts

- they are not comfortable interacting with operating systems in any depth

I don't have a big diverse data set, but the impression given is that if you expect this level of expertise you will be disappointed regularly. If the strongest CS programs pre-selecting for smart and driven students can't reliably impart that skillset, why would I expect other schools to?

For context, the original quote was:

* > How about understand the OS internals? How about write a compiler? How about write a library for their fav language? How about actually troubleshoot a misbehaving nix process?

Writing a compiler, writing a library for their fav language, and troubleshoot a misbehaving nix process are all examples of things I would definitely expect a CS major to have done at some point.

A SoTA compiler for Rust or whatever? Ok, no. But, you know, a compiler.

Ditto for library -- better than the standard lib? Ok, no. But, you know, a standard lib that's good enough.

ditto for debugging nix processes. Not world-class hacker, just, you know, capable of debugging a process.

I guess the other examples in that quote seem to suggest that "OS internals" probably means something like "knowledge at the level of a typical good OS course".

And who knows what those people meant by "comfortable interacting with operating systems in any depth". There could also be some reverse D-K effect going on here... "I got a B- in CMU's OS course" still puts you very well into the category of "understand the OS internals", IMO.

Ex-Amazon here. You are describing standard skills required to pass an interview for a SDE 2 in the teams I've been in at Amazon.

Some candidates know all the popular tools and frameworks of the month but do not understand what an OS does, or how a CPU works or networking and do not get hired because they would struggle to write or debug internal software written from scratch.

[added later] This was many years ago when the bar raiser thing was in full swing and in teams working on critical infrastructure.

Also, gate-keeping is not helpful.

This term is really getting over-used. The purpose of job interviews is to decide who gets to pass through the gate. It is literally keeping of a gate.

Software engineers, even the ones that are so superpowered that they :gasp: got a job at Amazon once in their life, can go an entire successful career without knowing how to use a kernel debugger, or understand iptables or ifconfig, or understand how virtual memory works.

Some engineers might need to know some of those things, but it is absolutely bonkers to claim that you could never progress past level 2 at Amazon without knowing such things. I know this because I once taught a senior principal engineer at Amazon how to use traceroute.

For many roles in Amazon (particularly the tens of thousands of SDE positions that will end up working with the JVM all day long), asking such low level questions about how OSes work is about as useful of a gatekeeping device as asking them whether white cheese tastes better than yellow cheese. And that's why the term gatekeeping is used.

Yes, if there is a nasty issue that needs to be debugged, understanding the lower layers is super helpful, but even without that knowledge you can figure out what's going on if you have general problem-solving abilities. I certainly have figured out a ton of issues in the internals of tools that I don't know much about.

Get off your high horse.

Playing devils advocate I guess it depends on what sort of software you're writing. If you're a JS dev then I can see why they might not care about pointers in C. I know for sure as a Haskell/C++ dev I run like the plague from JS errors.

However, I do think that people should have a basic understanding of the entire stack from the OS up. How can you be trusted to choose the right tools for a job if your only aware of a hammer? How can you debug an issue when you only understand how a spanner works?

I think there's a case for engineering accreditation as we become even more dependent on software which isn't a CS degree.

If someone is trying to debug that LB and doesn't know what a socket is, or debug latency in apps in the cluster and not know how scheduling and perf engineering tools work, then it's going to be hard for them, and extremely likely that they will just jam 90% solution around 90% solution, enlarging the frame to do more and more, instead of actually fixing things, even if their specific problem was easy to fix and would have had a big pay off.

Erlang is[0] what you can get when you try to design a coherent solution to the problem from a usability and first-principles sort of idea.

But some combination of Worse is Better, Path Dependence, and randomness (hg vs git) has led us here.

[0] As far as what I've read about its design philosophy.

Complex problems often have complex solutions, the algorithm we need to run as developers is - what's the net complexity cost of my system if I use this tool?

If the tool isn't removing more complexity than it's adding, you probably shouldn't use it.

I have written C and C++ for decades, deployed it in production, and barely ever looked at assembly language.

Kubernetes isn't a good abstraction for what's going on underneath. The blog post linked to direct evidence of that which is too long to recap here; I worked with Borg for years, etc.

Source, I was a Google SRE for 5 years (Ads, Traffic). I ran the in-house kubernetes clusters at a company for 3 years (so, no, no hosted kubernetes, we stood them up either on pretty naked VMs or bare metal).

I think learning AWS/kubernetes/docker/pytorch/whatever framework is buzzing is easy if you understand Linux/networking/neural networks/whatever the underlying less-prone-to-change system is.

You could also read some books. Rami Rosens "Linux Kernel Networking - Implementation and Theory" is quite detailed.

The "UNIX and Linux System Administration Handbook" (Nemeth et al.) covers a lot superficially and will point you in the right direction to continue studying. It's very practical-minded.

For low-level socket programming, you can probably read "Advanced Programming in the UNIX environment". It might be more detail than you need though.

At the other extreme, if you want to study distributed systems, you could read Steen & Tanembaums "Distributed Systems"

I'm totally self-taught and have never worked a programming job (only programmed for fun). Do professional SWEs not actually understand or have the capability to do these things? I've hacked on hobby operating systems, written assembly, worked on a toy compiler and written libraries... I just kind of assumed that was all par for the course

This manifested in my frustration when I lead building a new transport layer using just sockets. While the people working with me were smart, they had limited low level experience to debug things.

Professional SWE are professional in the sense that they know what needs to happen to get the job done (but I am not surprised when someone else does not get or know something that I consider "fundamental")

Pretty much any command I don't run several times a month, I look up. Unless ctrl+r finds it in my history.

All of these were still table stakes when I graduated from small CS program in 2011. I'm still a bit horrified to discover they apparently weren't table stakes at other places.

Any one of the undergraduates who take the systems sequence at my University should be able to do all of this. At least the ones who earn an A!

But developers should understand what assembly is and what a compiler does. Writing a library for a language you know should be a common development task. How else are you going to reuse a chunk of code needed for multiple projects?

Certainly also need to have a basic understanding of unix processes to be a competent developer, too, I would think.

I understand how a car engine work. I would actually explain it to someone that does not know what is under the hood. Does that make me a car mechanic? Hell no. If my car breaks down I go to the dealership and have them fix it for me.

My car/car engine is ASM/OS Internals/writing a compiler/etc.

Quickly getting up to speed on something you don't know yet is probably the single most critical skill to be a good engineer.

Once again: if you don't know your stack, you're just wasting performance everywhere, and you're just a code plumber.

isn't that why MTU discovery exists?

> Write your software to take advantage of the platform it's on, and the stack beneath it

sure, but usually those bits are usually abstracted away still. otherwise cross-compatability or migrating to a different stack becomes a massive pain.

> The simple fact of using http2 might change your organisation from one fat file served from a CDN, into many that load in parallel and quicker.

others have pointed out things like h2push specifically, that was kind of what i meant with the "(much)" in my original comment. Even then with something like nginx supporting server-push on its end, whatever its fronting could effectively be http/2 unaware and still reap some of the benefits. I imagine it wont be long before there are smarter methods to transparently support this stuff.

God knows how a UDP-based http is going to work but these are considerations a 'Software Engineer' who works on web systems should think about.

In 95% of cases if you want to get something/anything done you will need to work at an abstraction layer where a lot of things have been decided already for you and you are just gluing them together. It's not good or bad. It is what it is.

https://www.youtube.com/watch?v=ZSRHeXYDLko

Which says a lot about the situation we find ourselves in, I guess.

It is a process of commodification.

The problem with software development as a discipline is its all so new we don't have proper division of labor and professional standards yet. It's like if the people responsible for modeling structural integrity in the foundation of a skyscraper and the people who specialize in creating office furniture were all just called "construction engineers" and expected to have some common body of knowledge. Software systems span many layers and domains that don't all have that much in common with each other, but we all pretend we're speaking the same language to each other anyway.

I sometimes hate joke/fantasize about nailing a SE candidate with an obscure BPG or esoteric DNS question and then being outwardly disappointed in his response, watching him realize he’s going to lose this job over something I found completely reasonable to ask, but ultimately entirely useless to his position

Developers who can only configure AWS are software operators using a product, not software engineers. There’s nothing wrong with that but if no one learns to build software, we’ll all be stuck funding Mr Bezos and his space trips for a long time.

Huh?

This is absolutely fine, and one can draw parallels in software, as a mid level software engineer working in an AWS based environment wont generally need to know how to parse TCP packet headers, despite the software/infrastructure they work on requiring them.

Wait, what? Are you telling me that jet turbine blades are one single crystal instead of having the usual crystal structure in the metal?

https://www.theengineer.co.uk/rolls-royce-single-crystal-tur...

https://www.faa.gov/regulations_policies/handbooks_manuals/a...

"I don't know what to think when I meet engineers who know UNIX but don't quite understand assembly."

What you quoted is tantamount to the lament of a dinosaur that has ample time to observe the meteor approaching and yet refuses to move away from the blast zone.

Less facetiously: the history of progress in most domains, and especially computing, is in part a process of building atop successive layers of abstraction to increase productivity and unlock new value. Anyone who doesn't see this really hasn't been paying attention.

Can we provide an example that isn't also a big company? I'm not really thinking of big companies that don't either dogfood their own tech or rely on someone bigger to handle things they don't want to (Apple spends 30m a month on AWS, as an example[0]). You could also make the argument that kind of no matter what route you take you're "relying on" some big player in some big space. What OS are the servers in your in-house data center running? Who's the core maintainer of whatever dev frameworks you might ascribe to (note: An employee of your company being the core maintainer of a bespoke framework that you developed in house and use is a much worse problem to have than being beholden to AWS ELB, as an example).

This kinda just sounds like knowledge and progress. We build abstractions on top of technologies so that every person doesn't have to know the nitty gritty of the underlying infra, and can instead focus on orchestrating the abstractions. It's literally all turtles. Is it important, when setting up a MySQL instance, to know how to write a lexer and parser in C++? Obviously not. But lexers and parsers are a big part of MySQL's ability to function, right?

[0]. https://www.cnbc.com/2019/04/22/apple-spends-more-than-30-mi...

I know how to use it certainly, but how the hell it is implemented is more or less black magic to me.

Now that’s not to say I couldn’t learn how a socket works. It’s just never been at all relevant to performing my job.

[1] https://www.joelonsoftware.com/2002/01/06/fire-and-motion/