I am continuously hiring new people, mostly senior devs and tech leads. Maybe one in a hundred has any understanding of what virtual memory is. Or why one process can't access memory of another process and, if you really want it, how to set it up.
Two decades ago that was common knowledge.
It went away just as the knowledge of how networking works. Currently, if they try to open a connection and it does not work they are pretty much at a complete loss to what happened, where and how to fix it. Aside maybe from a simple problem like DNS resolution error (and not even that -- most devs don't understand how DNS propagation works, for example).
So if those problems happen, they are mostly deferring to me as a guy who can solve every problem. I see no clear path to achieving a goal of having someone else to learn to do the same.
If at least was able to purchease a home I wouldnt have this pressure.
I guess in the US is different because people have higher salaries.
You have to have the time/resources to dive that deep. Or maybe it's your job, line some of the folks in a local company that work in hard problems.
But even they are not that well payed.
Finding time to learn if this is not your day job is another problem, but a solvable one if you're willing to learn.
It's certainly much easier to find the time before you have to support yourself
It’s not too different from pro tennis players, who typically start playing before the age of 8.
For those at the top of the game, computing is a calling as much as a job. They will do substantial learning on their own initiative.
Your latter point is far more important to the matter. Those who treat it as a passion more so than a job, are more likely to be the trendsetters. Growing up and being free from responsibilities makes it easier for that thing to become your passion.
And let's also not forget, a few decades ago, computers weren't exactly a cheap thing for parents with little understanding to let their kids tinker with at will. Being born in a family with enough wealth to get a computer, enough wealth / understanding to let a kid tinker with it, was an immense boon. A long with anything that type of family tends to have going for it alongside wealth. It's not that far-fetched an idea that it's the other things, rather than the early age interest of the kid itself, that got them into such a position later in life.
I think it's important to keep things in perspective here - when I took my first programming job the first week was setup. On my team, our onboarding has you branch, review and merge on your first day. Those abstractions allow for people to focus on the areas they're working on, and iterate there. A game developer doesn't need to understand the complexities around js type conversions, any more than a js developer needs to understands rendering thread latency in a browser.
On every team I've ever worked on, that headspace from the team would have been much better served by understanding the product and project, IMO.
It's not about any particular detail, obviously.
He himself does not ever have had to previously see /proc/pid/mem before in his life in order to solve a problem involving it.
However many times a day he gets a question of any kind, that's how often.
also, devs seem to not understand that a computer network is basically a massive distributed system (on the routing level anyways), and that things inside this system can and do fail nearly constantly.
things like latency, endpoint connectivity and available bandwith is not a fixed given!.
So many devs in my experience seem to write code which assumes the network is always available with the same characteristics as the point in time in which they wrote the software.
Not to even mention devs are even entire companies building HA mechanisms which rely on ethernet connectivity to work, making it very hard if not impossible to stretch them properly across network segments/locations without stretching ethernet. (which brings its own set of problems).
Assume everything is broken, all the time.
When "the cloud" came out, I could easily understand the magic behind it all, so it made a lot of sense to me and it still does.
I feel "computer literate", however, even a lot of solid but younger engineers I work with struggle with lower level concepts and just don't have the knowledge and experience with things like networking protocols to be able to solve hard problems.
I'd love to hear from somebody with experience in scientific or other disciplines and IT who could contrast their fields with ours. I only know that I have had to go out of my way to pick up a historical component to my IT education, whereas it seems like a historical foundation is part of traditional math, science, and engineering training.
Vernor Vinge was right, though-- software (and hardware) archaeologist will be a real job.
And then there's the mix between the two in languages that rely heavily on powerful type systems.
I know I have read about it in books as a fiction, but I assume it must have already happened here before.
Maybe its an oxymoronic question to ask, seeing as if its lost we might not even know its lost, but more in the vein of "We put the lime in the mortar because this is what we have always done", unaware of the actual properties of lime when interacting with concrete.
Usually following the fall of empires. Ibn Khaldun (1332-1406) wrote:
"Perhaps they have written exhaustively on this topic, and their work did not reach us. There are many sciences. There have been numerous sages among the nations of mankind. The knowledge that has not come down to us is larger than the knowledge that has. Where are the sciences of the Persians that ‘Umar ordered to be wiped out at the time of the conquest? Where are the sciences of the Chaladaeans, the Syrians and the Babylonians, and the scholarly products and results that were theirs? Where are the sciences of the Copts, their predecessors? The sciences of only one nation, the Greeks, have come down to us, because they were translated through Al-Ma’mun’s efforts. He was successful in this direction because he had many translators at his disposal and spent much money in this connection."
Maybe not quite in the context you were asking, but philosopher Alasdair MacIntyre argues as the premise in his book After Ethics that this is what happened to the philosophy of ethics after antiquity.
Copypasting from its Wikipedia article [0]:
> [After Ethics] begins with an allegory suggestive of the premise of the science-fiction novel A Canticle for Leibowitz: a world where all sciences have been dismantled quickly and almost entirely. MacIntyre asks what the sciences would look like if they were re-assembled from the remnants of scientific knowledge that survived the catastrophe.
> He claims that the new sciences, though superficially similar to the old, would in fact be devoid of real scientific content, because the key suppositions and attitudes would not be present. "The hypothesis which I wish to advance", he continues, "is that in the actual world which we inhabit the language of morality is in the same state of grave disorder as the language of natural science in the imaginary world which I described." Specifically, MacIntyre applies this hypothesis to advance the notion that the moral structures that emerged from the Enlightenment were philosophically doomed from the start because they were formed using the aforementioned incoherent language of morality.
The original machine had a base-plate of prefabulated aluminite, surmounted by a malleable logarithmic casing in such a way that the two main spurving bearings were in a direct line with the pentametric fan. The latter consisted simply of six hydrocoptic marzlevanes, so fitted to the ambifacient lunar waneshaft that side fumbling was effectively prevented. The main winding was of the normal lotus-o-delta type placed in panendermic semi-bovoid slots in the stator, every seventh conductor being connected by a non-reversible tremie pipe to the differential girdlespring on the "up" end of the grammeters.
— John Hellins Quick, "The turbo-encabulator in industry", Students' Quarterly Journal, Vol. 15, Iss. 58, p. 22 (December 1944)
Anyways, that just to say that people in post-Roman Empire Europe lived out the reality of lost knowledge. They lived with the remains of incredible art, architecture like the coliseum, public works like the aqueduct, etc. But they wouldn’t have known how to reproduce those works. How to make concrete, etc. was knowledge that was lost to them.
The decline in big public works were probably the result of the loss of an empire that could cocentrate resources.
I thought concrete did continue to be used, but differently?
This all seems to spring from the myth of "dark ages"
The Antikythera mechanism comes to mind. It dates to the 2nd century BC. Nothing like it was constructed afterwards, until the 14th century.
The thing is, we tend to lose the knowledge that is deemed useless. Anything considered useful is well spread. The problem is when our opinion doesn't match the reality, we have something around a generation to prove it, or the chance is gone.
Mathematics is full of examples of things that were discovered again and again, until somebody found a use for them and they entered our textbooks.
Actual IC design (VLSI etc) is still a purely analogue field and all digital technology is fundamentally analogue at its core.
The reality in that sphere is there is little reason to limit oneself to the constraint of expertise in analog circuit designs when one can achieve the same functional outcome digitally, and use those skills more broadly.
Only where component sourcing is artificially limited, or risks of digital operation sufficiently large, and where the job market will support it, does it make sense to proceed in growing in analog circuit design expertise.
Meanwhile these designs are nearly all being functionally lapped by those in general industry, which participates in all the accelerating gains of digital technologies.
Much as I'd like that, I disagree.
The momentum has always been against that. We're members of a cult of the new, which does not value or even acknowledge the old. Which is why we keep reinventing the same solutions but with different tech stacks.
This drives developer value down which drives developer salaries down, so I don't see it changing.
FWIW, my history of math class as an undergrad was a math elective that almost nobody else took. Other math students didn't want to learn history and other history students didn't want math. And the curriculum stopped at the 18th century, before things got really interesting.
This just isn't true. We have Heartbleed because we refuse to move off how we wrote software 30 years ago[0]. HTTP was invented in the 1980s[1]. REST in 2000[2]. TCP/IP in the 1980s[3]. Ethernet the same[4].
I'm pretty sure we acknowledge and value all of those things. Our field, depending on how you define it, is only about 70 years old. Those things are pretty old in those terms.
[0] https://queue.acm.org/detail.cfm?id=3212479
[1] https://en.wikipedia.org/wiki/HTTP
[2] https://ics.uci.edu/~fielding/pubs/dissertation/rest_arch_st...
[3] https://en.wikipedia.org/wiki/Internet_protocol_suite#Adopti...
Some of this just seems to be due to the maddening popularity contest of technologies and frameworks within companies. There is a frantic rush to make sure you're doing everything that your competitors are doing, but not much introspection regarding whether things will actually help your company. You're just keeping ahead for the sake of keeping ahead.
I think today it’s probably more true than ever. Schools are under pressure to turn out grads who will be productive on day one and I think it’s shortsighted.
The problem was that there was a shortage of developers, and that most universities still only had CompSci degrees, not Software Engineering degrees, coupled with those degrees were 3-4 years long... So enter the code bootcamp, where you're basically given a hammer, and taught that every problem can be solved with it..
Now those bootcamps aren't all like this, and there were and are good ones out there..
...but from what I've seen (I want to stress that this has happened rarely compared to the rest) a graduate of one of faster bootcamps, that learned a single language, like JavaScript, and try and solve every problem with whatever answer they found someone else upvoting.. without any amounts of thinking if it's the right solution for this specific problem.
The problem is as your professor said. Do you want an education or training... And also as you said, that you can't turn out effective software engineers with a quick or short amount of education.
Is a Software Engineering degree really a thing? Perhaps outside the US? Can you point to a notable university that has this major? A quick search gives me the impression this is an online correspondence program type of thing.
At UCSD in the late 90s we had a Software Engineering class but that was it. Otherwise internships were supposed to prepare you for your careeer.
They do have a full SWE track as a Master’s program, which I did, and quite enjoyed.
Penn State has one.
I cannot stress how important my education was to my current understanding and success as a software engineer in the field. A poor education is a huge detriment. When I joined the workforce I found many colleagues without one ill prepared for some of the basics, like debugging kills, using tools outside their domain expertise, etc.
That said I don't think a formal education is REQUIRED, but without one you need a certain kind of perseverance and thirst for knowledge not everyone has. I've had the pleasure of working with several folks in that category, so a formal degree isn't for everyone.
Today, it appears specialization is happening much earlier in the education process without a general foundation. Specialists will always be needed, and larger orgs can get by just fine with exclusively specialists, as long as their communication skills exist to prevent silos. Larger companies can afford this. However a good generalist can replace a team of specialists when the latter budget is out of the question; a huge win for smaller orgs, start-ups etc.
The old joke that a good generalist is the "jack of all trades, master of none" has some truth to it, but as with all things, it's a tradeoff. A team of qualified specialists will definitely produce better work within their domain expertise, but at a cost that can be prohibitive when "good enough" is within reach.
One wants to go straight into the work market, with focus on getting the bases, technical school.
One wants to go into the work market, with some more preparation than only the bases, polytechnic institut.
One wants to go into the work market, with overall experience, in a degree certified by the engineering order as qualified for professional title certification exam, university.
Unfortunately not all degrees, or universities are well prepared to offer this, or have staff that cares about making it happen.
Spend some time in the coin op game community... This is all we spend our time on. Well that and cabinet repair.
But seriously though, keeping these old games alive requires serious reverse engineering of both hardware and software. Especially if they have custom chips, ASICs, undocumented features, etc. But the work has to be done or these games will be lost to time... It may be a passion project/ hobby, but it's still real work.
I like your point about digital archaeology. If it's important enough to some people to keep old arcade games running, how much more important is it to understand and keep some old mission critical computer running.
It's a lot like the tech people you see in these dystopian future movies, the ones who are like "oh yeah you have an old z80 with the qxp interface, haven't seen one of these things in years". Like Scotty from Star Trek. They're an increasingly important part of the tech ecosystem.
Besides, the PlayStation based hardware of this specific arcade PCB is not that far off from the ubiquitous 32-bit microcontrollers that underpin modern digital society; in fact the MIPS R3000 derived instruction set even bears a striking resemblance to that newfangled RISC-V thing every investor is talking about. And yet most of my 19-year-old university classmates want to stay away from "scary" embedded or kernel work in favor of web or game development.
It seems that with more abstraction, more people throw up their hands and say "I'll never understand it", "The designers of this system did things in the stupidest way possible", "It's magic" or (in the case of the financial system) "It's a rigged game" "You're working for the man"
What all of these points of view have in common is that the people who hold them have given up trying to understand the world around them. Instead they take the intellectually lazy way out and compose their own pet theories of why things are the way they are, this is where tinfoil hat stuff comes from. This is dangerous, and is sort of the point of the original article.
I keep telling people that "the world is still knowable, still understandable". You have to put in the time and learn about it. It's easy to do, you just have to do it. There are plenty of "average" people playing in these "difficult" to understand fields.
What we desperately need to cultivate is a culture that encourages and rewards curiosity. When people are curious, they take the time to learn about the world around them, and why things are what they are.
If you don't master the world around you, it masters you.
During the crypto boom I distinctly remember hearing someone say something about creating a decentralized Discord. I was like... that's IRC.
Nobody said that. But spending a week is definitely an option to all programmers, and many would benefit from it.
https://training.linuxfoundation.org/training/a-beginners-gu...
A motivated person can run through it in a couple weeks while working a day job.
I know this is tired and cliche at this point, but literally this week I sat down with ChatGPT and asked it to teach me how to write WAT (web assembly text format) so I could understand how memory is managed at a really low level (but not so low that I risk crashing my computer).
It turned out to be super valuable. This is where AI shines for me - I can ask it any question that pops into my head, and also validate whether what it's telling me is true by running the code and seeing whether it works. It was amazing.
If you're curious, I'm fine sharing a link to the chat:
https://chat.openai.com/share/583bf23b-b43d-4566-956a-e92b6f...
I’m sure it’s not error-free, but the speed of learning makes up for it IMO. Like I’d been struggling for a while to learn tree-sitter, the documentation is overwhelming. I had a chat with AI and got a working solution for my problem in probably 4-6 hours, and now I can write tree sitter grammars without help. It’s really incredible.
His book Code is fantastic. He starts at simple battery and lightbulb circuits and builds and builds towards a simple CPU.
He also wrote The Annotated Turing which is a breakdown of Alan Turing seminal paper and you only need high school math to get through it.
When I was in school my favorite course was compiler design and we used Compilers: Principles, Techniques, and Tools (aka the dragon book). It’s one of the best textbooks I’ve used but that was 30 years ago. There might be something better now. Understanding parsers and lexers and (especially) state machines is something that will serve you well.
Ah, I'm interested in parsers these days (I just wrote one for parsing org data, I am a bit unsure about the architecture of the project but at least it does what I needed personally). I am right now at a friend's place that just showed my the dragon book from its shelves, that's a funny coincidence. I will check out Code.
Python, Linkers, and Virtual Memory by Brandon Rhodes (Python core dev)
https://www.youtube.com/watch?v=twQKAoq2OPE
(Most of the content is not actually specific to Python)
He beautifully pulls back the curtain on so many lower level concepts like virtual memory management, dynamic linking, heap/stack, fork(), copy-on-write.
The talk is broad in nature, not deep. It takes you just below the surface of many magic black boxes, and, as you put it, enhances your proximity with those topics.
For me, so many things clicked in this single talk:
- How virtual memory works (incl. paging in/out, swapping)
- Why there's those discrepancies between RSS / PSS
- What segfaults and page faults are
- What actually happens when I get errors related to dynamically linked libraries, either at build time or runtime
- Actually understanding the output of top / ps
Godbolt [0] is an invaluable resource. But simply setting up tasks to yourself and completing them may be the best course of action. Then you'll find whatever resource you need for a concrete objective.
For example, if you have a week, I'd suggest to start "in the middle", and move up and down according to your tastes.
- Write a hello world program in C, compile it and run it. Try to compile it statically and understand the difference.
- Ask the compiler to produce an assembly file. Try to build the executable from that assembly.
- Try to write a hello world in assembly language by yourself, compile and run it.
- Write a program in assembly that prints fibonacci, or prime numbers.
- Now, forget about assembly and move upwards. Download python source code, compile it, and run a python hello world with that interpreter.
- Look at the Python source code, which is written in C, and try to follow what happens when you run your hello world.
- Try to change the "print" function of the python interpreter so that it prints your string backwards.
Depending on your experience, you may need more than a week (5 days) to complete all these steps. But that's OK! In a few months you can find a new spare week and continue your work.
To build mobile phones, we need a lot of tech built on tech build on tech. Dwarfs on giant's shoulders. And there's a place for people who don't understand the underlying tech. I've got a colleague who doesn't understand any of it, yet can do useful frontend work. But when you stand too high above the ground, the problems in the article become real, and that will get us stuck.
Programmers at the time went from thinking these kit computers are toys to watching all the old ideas like virtual memory, cache, pipelining, networking, and multiple cores get reintroduced as the anemic transistor budget exploded. None of it was particularly novel, but the slow introduction of older ideas was a great way to learn all this stuff from the ground up.
However, if I am using an abstraction like an ORM without understanding what is happening with the database under the hood, or why it's getting slow all of a sudden, it is bound to bite me in the ass.
I shudder to think of how much money and energy the world is burning every second just because of this one alone.
It can take days for them to even track an evil SQL query back to the actual code. There were fetch-one calls that literally carry a million rows back over the wire for that one returned entity. Indices get skipped due to the magic translation of a row.col.toUpper()=="STRING" into SQL.. when the db collation is case insensitive in the first place.
But hey, the devs don't need to know SQL, so there's that.
Specialization always exists in an industrial society.
And web development has the worst tech abstraction of all.
There used to be a time where any Software Engineers ( i think the term wasn't even invented then ) or simply programmers would know at least a thing or two about Hardware. We now have so much abstractions we have people working in tech who have zero knowledge on either software or hardware, or any low level stuff.
That is why not only do we need some open source or open standards ( I am not a zealot for everything open sources ), we also need to simplify everything we have today. To basically refractor everything we have learned from Hardware to Software.
They know only full SPA frameworks, they have never seen a dump of an HTTP message, headers and verbs are abstract things to them. Hell, many of them don't know you can have fully functional websites with zero JS, including payment, video, login, etc.
I started to write an HTMX tutorial (https://www.bitecode.dev/p/a-little-taste-of-htmx-part-1) because I noticed a lot of young coders don't understand what to do with it. They read the tweets saying it's nice, but when they look at it, it makes no sense to them.
It's really fun because I now remember how some senior coders looked at me, knowing nothing about compilation. I was struggling with Python packaging because before wheels, it required compiling a lot on linux, and it failed often. For them it was obvious: just install the headers, look you need the dev packages, wait, you don't have gcc?
Nowadays I happily patch nginx source code and compile it manually, but it took a lot of work to learn a minuscule chunk of what all those guys knew by heart.
Because they started with assembly.
Around I guess 2017 or even 2016? I used to think this is some sort internet troll comment about people never seen a dump of an HTTP message, until......
>Because they started with assembly.
It wasn't necessary starting with assembly or something low level. ( Although that certainly helped ) We have less entertainment, more time, and no Internet ( for most part ). Things also dont work all the time. And we have to spend time figuring it out ourselves. That is where all the knowledge comes from.
The difference is where you start figuring things out.
Before, you needed to figure things out at your level, because it was the immediate area of mystery.
Now this level is generally solved, you need to figure things out at a different level:
- filter out the mass of irrelevant information, the out dates one, and the one from spam
- understand how all those complex abstractions interact. We have good resources on how each work individually, but the carthesian product of the monsters we build with them, we obviously don't
- debug some things that don't work when the magic fails, way below your level, or behind a service, where nobody is looking
The problem before was scarcity of info, lack of standardization and roughness of systems.
Now it's abundance, opacity and too much sophistication.
But everybody still have to figure things out. Just no the same challenges.
I'd argue that the "find and synthesize" generation have an advantage within contemporary software paradigms because of their experience but, without deep knowledge of the foundations they are building on, they might be disadvantaged when it comes to imagining/creating new paradigms.
Then again, first-order thinking seems to be easier when you're not marred with the traditions and conventions of the past so maybe this isn't actually a disadvantage.
Finding and synthesizing doesn't do you any good if you're unable to meaningfully apply it or understand what you're applying and what you're applying it to.
Trial and error also doesn't do you much good without the ability to back it with knowledge and to find the relevant knowledge.
This makes valuable anyone with long attention spans and immunity to boredom.
For me, it has taken some habit forming (and habit prevention), but I have learned so much in the past 10 years since giving up gaming and social media. HN is an occasional vice though...
When I was teaching programming, I had a fun party trick whenever we got to HTTP. I'd fire up netcat (in listen mode), then connect from a web browser and "serve" a website by hand. I'd show the students the HTTP request that came in, and just manually type out a simple HTTP response and they'd see it appear live in the browser. Its pretty magical.
And once I'd shown them that, I'd write (by hand) an HTTP request to wikipedia or something to show them how its symmetrical.
Of course, real websites increasingly do HTTP2 over TLS or something. So unfortunately its not as "real" as it once was. But if wide eyed look on my students faces is anything to go by, it was a great lesson.
The email telnet thing was also a good learning experience. Gmail's servers are fun because you can see the designer's cute messages; if you forget your EHLO, they'll throw an error (or at least they did years ago) that it's polite to say hello first.
This is not true. I run a mail server for myself and some friends. It started as an old desktop running under the desk at a university, transitioned to the back corner of a server room when I was working as a network engineer, and now it runs on a Raspberry Pi in a closet of my house. I had to pay extra to get a static IP at home, but everything has been off the shelf and DIY. My current Raspberry Pi has been running for nearly ten years, with only one interruption (the SD card failed). The idea that you can’t run your own mail server is a myth, and I think more people should do it. It is not hard, and you will learn a ton.
(I used to run mail for a large corporation, so yeah, I know a lot. But I learned how to do it by running a mail server out of my dorm room.)
And indeed, the first time opening a POP3 session to an email provider through telnet was also an amazing feeling; maybe even more visceral thanks to its "live coding"/immediate feedback aspect; but personally just a small bit less foundational. So, anyway, through this whole story, what I really wanted to say is - thank you for your service and approaching your teacher's post in a great way!
Unfortunately chrome has stopped showing cookie headers in the network tab. You have to look somewhere else for those, and I think at that point you’re just seeing the current cookies, not the raw headers. Maybe there’s a security reason for that, but it is a further abstraction layer of “we’ll present the information a certain way” rather than “we’ll just show you all the information that the server sent back” that I expect will grow into other headers (maybe already has) and further remove us from seeing the raw details and having a clear understanding of what’s happening under the hood.
(You know it’s possible HTTP/3 cookies are even passed in a different way under the hood from other response headers, and that could be part of why they’ve separated them out. I hadn’t considered that possibility.)
https://www.networksfromscratch.com/
It will take me 10 years to finish it at the pace I’m going though :)
And for once in my life, I'm actually going to sign up for your email list because I want to know when this thing is done!
Sure, a frontend dev who strictly works on that won’t know much about, say, the OS layer. But they don’t have to, it’s not part of their job in any way.
There is no shortage of young people that work on areas that makes them require deep knowledge of many different layers of the stack, it is simply not necessary for every IT-related job. But one can absolutely pick it up if they want to.
The HTTP protocol, HTML and vanilla JS are not "the OS layer", and yes, it's part of your job as a web developer to at least understand the basics of them. There are many so-called "frontend devs" nowadays who literally only know React, and if you asked them to create a basic webpage where clicking a button changes some text in an element without React, they'd be completely lost.
Usually the problem with this kind of developer is not just that they don't know, it's that they don't care. They WILL inevitably run into problems that require this basic knowledge to solve because the frameworks can only hold your hand so far, and instead of trying to figure out what's going on under the hood that's causing their issue, they will instead shrug and go "it's not part of my job" and just write a bunch of garbage spaghetti code in an attempt to work around the issue.
Pretty much anyone who has been around long enough has met someone with years and years of experience doing exactly one thing, whether that's WordPress or java CRUD or something else where they have blindingly obvious gaps in their knowledge that someone with their experience shouldn't.
The lack of deep knowledge isn't limited to a single field.
This is an entirely new business domain, it’s not really trying to be the same.
No wonder a lot of modern software is terrible.
To be fair, the last time I "opened up" a computer and changed its RAM was about 10-12 years ago, and since that time I've only worked on a Mac Mini and on a laptop meant that "opening up" computers became a thing of the past.
People here have way too much confidence in their interview questions being a good signal for experience. It’s pretty wild.
You are proving my point. You should care.
My point is that your question is some esoteric gotcha party question that may as well be out of Trivial Pursuit. But you’re treating it like anyone who doesn’t care is just being a bad engineer. There are countless ways engineers spend their time, and choosing what to work on is the most important choice of their careers. It’s on you to justify the claim that knowing the difference between a JS form submit vs browser submit matters at all, let alone that it’s a distinction that comes up in day to day life.
For you that would be trying to work in ML not understanding any of the theory or how it works, but having used OpenCV with some premade models a few times.
For a frontend web developer who, as a large part of their role, will need to communicate with backend systems… that’s not understanding how their FE web application actually communicates with BE systems.
And from experience… yes, this is very common. And it has a noticeable impact on their effectiveness. Trying to debug why some interaction between your application and the BE application isn’t working while thinking the dev tools network inspector is just black magic and nonsense makes your job substantially more difficult.
This does make these people bad engineers. They are not able to understand and solve a huge class of the problems that they face day-to-day and instead (in my experience) often fall back on “just try a bunch of different things until something works for reasons I don’t understand” which is a poor way to approach work and leads to overly complex, buggy, brittle systems.
I think you do. GP just phrased it oddly.
The distinction is between a traditional HTML form and browser page load, vs XMLHttpRequest.
Ultimately it's all HTTP though and on that level, there is no difference whatsoever.
I've started nearly every position in my career in a different domain without prior knowledge, including workflows, protocols and the languages that implement them. I may be an extreme case, but we exist and your process inherently excludes us. I've worked in robotics, consumer electronics, healthcare, fintech (not even in that order, although that probably would have made more sense.) I've delivered in each domain as well as my expert colleagues. But if you asked me to implement a LLM, I'd shug and tell you I don't know how, but I _would_ explain my process for HOW I would learn to.
(Edit: typo)
I do get your point but depends on the role you are hiring for. I have had candidates get upset at me (for example: bootcampers) because they couldn't explain how to submit a basic HTML form but wanted me to look at their github "portfolio" that was done during the bootcamp with React/Express Code and what not. Their title was "Front end React Developer". That is a problem and I usually wouldn't hire those people.
For example, I would take a react developer with obvious gaps and a strong willingness to learn on my back-end services team over a passable back-end dev who was set in their ways (unwilling/uneager to learn new tech, entrenched opinions stated as fact, etc.)
Naturally, my approach is not a one size fits all situation and a great deal depends on org structure and mentorship opportunities being in place for it to work. The benefit being you avoid monoculture "silos", and have more cross collaboration and transfer opportunities between teams (some may call this "full stack", I wouldn't)
Several high performers on my team are "self-taught", work comfortably across several languages today and can pickup new tech easily. They came in knowing their "one stack" at the time. If they were "bootcampers" or not seems irrelevant and somewhat reductive/offensive.
That said, if the candidate in question shows _no_ understanding of their problem domain, can't reason their way out of a paper bag, and get visibility upset from questions when you try and tease that out, that is a certainly a red flag in my book.
With respect to your example given, however, it could be argued that http details because of the abstractions in place inherent in react, aren't critical domain knowledge for what is essentially a UX dev. If they can ship an experience your users enjoy more than the candidate that can recite the HTTP 1.1 RFC and cannot, then what have you gained in hiring the latter other than maintaining a culture of pedantry?
TL;DR it comes down to how much risk your team is willing to take on, as there's always risk inherent in hiring ANY candidate, including those that don't already check all your boxes in regards to tech know-how. I'm merely stating that by overlooking candidates that don't fit your self-imposed mold, you're likely missing opportunities for rewards that can pay dividends when it does work.
I rarely do any of that, usually my abstractions are container layers, a libc, and then a kernel between me and registers and dma. But that is a lot more layers than you'd think. I can't even understand the boot process on a modern machine with tpm. And i have no clue how many processors or controllers are in my computer. Every usb controller has an entire arm core in it, mice and keyboards have 8051 running c-ish code someone somewhere wrote, there is no bottom to the complexity.
Similar with mediatr, I know conceptually that it "just" checks for the right classes in the loaded assemblies. But still feels like some weird incantation to use, to me. And I have such huge knowledge holes across all layers.
Sometimes i feel my lack of knowledge to such a degree, that I question whether I should even be a programmer.
This wasn’t in like 2010 when Firebug was just coming out and people still used Firefox as a rule, this was like 2015 when dev tools was in every shipping browser approved for corporate use by a technology laggard company.
So the moral of the story is just blame DNS first.
I'm about half a decade into my career and I've recently tried to take note when I hit milestones or achievements, even little ones.
I was helping an intern with a tool and it didn't have support for what we needed it for. Since the tool was open source, I just cloned it and patched it to add support for what we needed. When I told the intern this, he couldn't comprehend that I was so blase about modifying this tool that seems like witchcraft to him.
But what I quoted, I feel deeply. I've worked with so many people with so much knowledge they can't possibly communicate it, and I'm only starting to really understand the tech around me. I didn't think anything of patching the tool at the time, but its important I look back so that the younger version of me can see the progress I've made.
A lot of CS skills are generalizable. Knowledge is one component you pick up with time. A good education, self-taught or otherwise, should allow you to drop into any sort of code and be effective without much spin-up. College covers CPU architecture, assembly, networking, operating systems, web, algorithms. This is not esoteric stuff, it is very standard and you can get it all in class or from textbooks free online!
It’s tough if you feel a degree of responsibility for their success. Mentors are one of your greatest assets early on (and arguably later as well), and to try hard to have them succeed and thrive only to see them languish on trivial tasks is awful.
I think part of the problem is that CS education where I live is awful. The kids come out of school expecting real work to be wildly different than it is, and it hits them like a brick wall.
Today, there are probably a vast number of entrants who heard that CS is the ticket to a high paying job, and they are also told that an internship is a vital bullet point on their resume, if not a guarantee of a job at the internship site.
Then, as now, students studied under the constant drone of "you will never use this stuff once you finish college." They still have to decide if they're actually interested in the subject matter or not.
A good bellwether of career interest is the students in the youth symphony. They've all aced every subject in high school, plus rocket club, gentleman sports, and orchestra. The program for the end-of-season concert will have a little bio for each graduating senior, including their college interests. Half of these kids want to major in CS.
Once you have the latter, either by having built such a codebase, having worked in one, or even having experience with puzzles or games requiring multi-step planning and understanding of the potential failure points at each step... it's absolutely transferable. But there are also a lot of people in our industry who have memorized interview questions and see their role as churning out components. And while arguably that's not a CS education, it's being called the same, and it does a disservice to those people's careers.
I graduated with a Computer Engineering degree, did assembly, C, microprocessor design, computer vision, and know a good bit about lower level stuff, how memory works, how networking works, etc. All the stuff people in this thread seem to be lamenting the lack of. But I was also a shitty employee fresh out of school because I didn't know anything at all about modern software development because there was absolutely no time to learn that stuff as well.
I still had to spend a lot of time getting good before I was worth anything, just as these "new frontend devs" will, as well.
Yeah, but modern software development is trivial to learn, particularly in comparison to a computer engineering degree. You see "developers" here on HN gloating all the time about how they didn't need any post-secondary education at all to get their jobs; these frameworks are literally designed to be usable even by minimally skilled coders. You were in a much better position having to learn modern software development after a computer engineering education rather than having to learn the rudiments of computer engineering on the job after getting an education in modern web development.
LMAO c'mon. This basically undercuts anything else you say here. Go teach your grandma react and see how trivial it actually is.
It's more likely that I will do something like "a full web app from installing the OS on a fresh laptop to coding it to hosting it on server".
It would also be more beneficial: plenty of tutorials for individual techs, rarely they teach you how to integrate them together and go beyond the toy example on localhost.
But for that I need a better platform than substack. Therefore that's not going to happen anytime soon.
I agree with you but HTMX? That's a big abstraction layer. A good one but still, not really a way to avoid layers of abstraction. Pure JS makes more sense.
We work with both high and low level tools where I work, and in my experience there isn’t necessarily a lack of understanding among younger developers. The CS courses still teach you a lot of the basics, and to actually use what you call “hype” tools efficiently you typically need a rather good understanding of how they work.
I actually don’t think we have enough abstraction, especially not in the DevOps field. In my opinion we’ve overcomplicated the deployment procedure without making sure it was abstracted. We don’t need our software developers to know how our Cloud Networking and Virtual Networking works, we don’t need them to spin up resources and make various internal DNS and Firewall rules, we simply need them to focus on what they’re good at and then hand off the container to the operations department where people actually specialise in that knowledge. I’m actually fine with infrastructure as code, but it needs to be templated to the point that our developers simply “order”resources based on whatever template they want to deploy so that they don’t have to keep up with the constant changes or learn how networking in enterprise organisations work.
As far as the CPU stuff goes. Young developers do a lot of low level code. We write a lot of embedded software for our solar plants, and a lot of the time, they seem far more capable and “modern” in their approaches than the “old guard” exactly because they’ve been taught the same curriculum but also all the lesson learned by the “old guard” along the way. That being said, it’s not really necessary for a lot of developers to keep an active knowledge of how an x86 CPU works, because it’s very unlike that they’ll have to work with it. It will often be far more useful to know how various types of ARM processors work, as that’s something a lot of us actively work with.
I am afraid that many businesses prefer the vision where the software developers do the operations too, and the company can fire the operations department. Consider how much money you could save... before it all falls apart.
Short term optimization is how managers get their bonuses. The trick is to move on to the next project before the old one falls apart. Then it becomes someone else's problem.
It is basically idiotic to me, as a highly paid senior application dev.. that they basically want me to spend any cycles on stuff that would ordinarily be done by someone making 1/5th my TC.
If I can do it more than 5x as fast/efficiently, sure, but I don't. IaC is something I worry about at the start of a project, as hamfistedly as possible, probably over-provisioning so I don't have to go back to it anytime soon. All so I can move onto what I am paid to do - delivering functionality to stakeholders. Someone who deals with IaC as part of their full-time job will be the expert who can move more quickly & correctly through it.
Sometimes this is just galaxy brain budget arb, where the infra org gets to show a cost save, to the detriment of the appdev org.
Did you mean for this to be condescending towards ops? Because that’s how it sounds.
Ops is not easy, at all. Even if you’re doing zero IaC, I defy the average dev to try to provision a Linux box from scratch and install, configure, tune, and maintain the stack you need to run your code. Throw IaC atop that, and now you need to understand declarative programming and OOP concepts to be able to do it efficiently. Then, there’s K8s…
Not to mention the architecture side of things. Devs love to grab whatever shiny thing they saw on a Medium blog, even when it’s a poor fit to their problem (or their problem is unoptimized code). You probably do not need a columnar DB, you just need to normalize your tables and learn relational algebra.
Ops is absolutely hard and theres a lot to learn. And in doing so you generally may end up with DevOps teams with minimal to zero domain knowledge in what the company delivers to end users.
So if you have a team of people who built a deep level of expertise in something over 10, 15, 20 years and that something is business facing, domain knowledge, etc.. then it is not of value to have their time spent on other tasks.
Often these DevOps roles are filled by more junior staff at the earlier stages of their career. This is another reason the staffing is usually lower cost.
Hint: I wrote operating systems for fun before becoming an SRE. Imagine how much fun I have explaining context switches and CPU cache invalidation and their implications on their app’s performance, to application developers who consume frameworks and look down on my salary! Building a distributed computer at scale is much, much, much harder than your Linode tutorial expectations of what ops does.
Native software is mostly dead, so operations is software engineering now, at least those parts of software engineering you folks threw out when Docker came along and turn your nose up at, and I make more money every year cleaning up the low-hanging fruit y’all leave around, so…
Now take all the stuff you are an expert at, and imagine the CTO assigned to you - build dashboards of accounting information. You don't do UI, and you don't know accounting (maybe YOU do, but your average SRE/DevOps do not).
It wouldn't make sense right? And a lot of those tasks can be done by a junior "data analyst / BI dev". So it would be an expensive mistake as well.
This is my point.
The worst problem, which TFA alludes to, is that those "layers" are not really needed, they are leaky kludges upon kludges to make some lower layer unsuitable for later needs more palatable or able to handle some unforeseen use case that's against its design. Other stuff is just added as ways to sell new enterprise tooling, support and consulting.
Seeing the layers and technology stacks being added over time can give the impression of watching some organic evolution happen, when it's often merely accumulation, rough patching, and corporate attempt to push its technologies/NIH.
But it only works so far and at some point the application becomes unmaintainable mess and the effort to rewrite it from scratch starts.
Only we can't rewrite the world after it has become so unmaintainable that nobody can figure out how to change a detail of some layer inbetween the program and the transistor.
Heck, we are already stuck trying to fix the bad design of IPv4.
It reminds me of:
https://github.com/alex/what-happens-when
and how many of today’s CS-degree holders would barely understand any of it. As someone who has also “grown up with all the technology”, I’ve learned and experienced all that. But as a percentage of “software engineers”, there’s fewer and fewer that do every day.
Under our current system, knowledge that cannot be monetized is useless. Worse than useless, in fact, because time spent learning low-level concepts is time that could be spent learning marketable skills.
The industry is effectively paying us all to forget.
Sure, learning the USB stack and differential signaling is fun, but unless my job involves implementing custom USB devices from scratch then it's pointless trivia that won't pay the bills.
I suppose that is what will keep happening. At some point, someone will decide to re-write everything from scratch and re-discover lower layers of abstraction or challenge them.
A funny possible history is everyone forgetting what’s necessary and then having the relevant information they need accidentally blocked by an AI system they have no idea how to build or fix (maybe because of terms like master/slave going against elite values).
Not completely. Curious people will always pop up, it's just that they will take as much time as you did to learn everything.
I like to think of myself as one of the curious people. Ever since I was a kid, I had a huge desire to understand how the hell does this magic box called a computer work. Decades later, as a software engineer, no matter how much I learn, no matter how deep I go, there's always this voice in my head going "but WHY? WHY are things the way they are?".
I did start top-down instead of bottom-up, starting from the high-level languages running on modern operating systems, going more and more into the (professionally unnecessary) low-level through pure curiosity, but I still believe I will get to the bottom one day.
As a counterpoint, I'd say that it's also chance to build on top of these abstractions without having your brain cluttered with how they're implemented. That's also how science grows.
And don't underestimate the new players. They're perfectly capable to understand the low level details if they have to, and they have much more resource available to learn too.
I just hope when that time comes I'm well compensated for studying erudite texts on low level computer science while chilling on my phone instead of browsing social media or playing games. But honestly, I enjoy reading and learning about ABIs, C programming, network protocols, etc... more than mindless scrolling anyway.
Like you, I enjoy learning techie stuff, but most of my friends don't. Because, IMO, most websites are written for younger people, they feel almost helpless.
Free markets don't solve every problem but they solve this one.
I'm gonna guess this same message, in different words, has been repeated often throughout history for as long as people had the ability to reflect on their life and the next generation.
So really, what's different this time? What's the difference between forgotten technology that nobody needs and an unpublished author who's works were forgotten? I think the answer is none. We just think it's more important because this is the industry we care about.
We're actually really good at reinventing the wheel in this industry, so even if some software is forgotten, you can probably bet it'll be reinvented or reverse-engineered if it's really needed.
While we should preserve technology knowledge (because it's pretty cheap to do so), in terms of the economy, sometimes things are a stepping stone and should be forgotten.
What's the difference? Nobody needs to know how to make or operate a Chappe telegraph because they aren't used anymore. If every one that still exists suddenly vanished, probably no one aside from a few museums would even notice. But all that low-level stuff is still there at the heart of every computer.
There will still be people at e.g faang/semiconductor companies
that do this stuff day2day and even push advances in those areas and have already created solid training materials for younger employees
If you're worried that this knowledge will disappear, then feel free to write it up in form of tutorials.
It was around the year 3-4 mark that I decided to knuckle down and try to improve my fundamentals (data structures, algorithms, memory models, concurrency, CPU architecture and some network fundamentals) by reading popular papers and literature and writing all of my personal projects in C and C++. I’m about two years into this study and while it’s been immensely rewarding, I’ve found it to be a huge undertaking while juggling life and a full-time job.
What I’ve also noticed is that, while I understand a lot more about what the CPU is doing, memory manipulation and how to write more efficient programs, I haven’t found it be particularly beneficial to my daily work (still Python and JS). I would love to be able to put these concepts into practice for many hours of my working day but it’s difficult to move from general web-stack development to more performance-oriented development (embedded, low-latency, OS, etc.).
My guess is that this is one of the reasons we have ended up in this situation. You can get away without knowing the fundamentals (a good sign of progress?) and that if you really do want to pursue these areas that promote building this kind of knowledge as part of your career, the barrier to entry is quite high and the positions are fewer than say a decade or two ago. I find it a shame because in my eyes, these areas are the most interesting and exciting areas of programming. It’s an art.
I think this is because of the effect OP is describing: no matter how much time you spent studying, the odds that you learned about the specific thing that will make the difference in your work this week are slim because the technology is so broad and deep. Odds are that what you choose to intentionally study is irrelevant to your work.
You've picked a slice to learn about, but those who were working on it as it got layered know it all. Once you have that kind of knowledge, the odds of being able to explain some unexpected behavior approach 100%, but acquiring that knowledge took decades.
I've found a method that does work pretty well is to dig really deep into the topics that come up in your actual work. Instead of a random sample of fundamentals, find the parts of your job that feel like magic and explain that magic. Skipping over V8 and straight to the CPU might not be useful for a JavaScript dev very often, but a deep understanding of V8 is relevant more often than you might think.
There’re huge areas of the software industry where performance matters to this day. Examples include videogames, content creation, video processing, engineering software, science related HPC, and now AI.
However, transitioning from web development to these areas gonna be hard. That software doesn’t run in web browsers, the code is either desktop apps, deep inside web backends, or supercomputer programs. And high-performance C++ is often not enough for them, depending on the area ideally you might also need GPU graphics and/or GPGPU compute.
I was lucky I have never worked on web apps. I know TCP/IP reasonably well, I have some general understanding of HTTP and HTML, but I’m totally clueless about the modern frontend technologies.
Find a chunk of code you want to optimize, simplify it as much as possible, then put it into godbolt. Match the interpreter version to yours. Then go look up the bytecodes and see what they do. Try changing the Python version to see how it differs, or a different approach to the problem (e.g. a map instead of a list comprehension).
This takes an enormous amount of time, of course, but you can learn quite a bit.
[0]: https://godbolt.org
But we don't need every programmer to understand low-level kernel stuff, precisely because the kernel abstraction is good enough at hiding that stuff. Most of us don't need to care (whereas 40 years ago, many more did need to care).
Personally, I myself never want to care about low-level kernel things. When I'm trying to write a program to do something, if I can write it without having to worry about low-level things, that's a win.
For example, I am NOT a cloud expert, and barely a novice, by any measure. When dealing with various CloudOps/Cloud Architect/etc types, without fail, the good ones are also complete Linux gurus who would be at home architecting on-prem infrastructure as well.
Unfortunately there are a lot of "experts" who are only comfortable working in abstraction and so as soon as we have an issue that scratches below the surface, it turns into an all-hands-on-deck mayday situation, looping in other teams/experts to try and save them.
I've also seen horrific implementations by abstraction enjoyers.
Imagine a continuous integration environment of the following - k8s where one pod simply runs a forever shell script that is basically "git checkout master;git pull; sleep 300;". Then the other pod actually runs the application on top of the shared storage that the first pod writes to. This script, unsurprisingly, fails silently and hangs frequently. To debug this you need to go through some cloud auth portal and click through some web UI to then open a console in your browser (which doesn't support copy/paste).
The idiot version of doing this on-prem would be a single small VM that has a cron running every 5 minutes, which invokes your deployment script, with cron configured to email on failure. This would be simpler, more reliable (not relying on a forever script), and more supportable (it emails on fail!). And it uses like 30 year old tech that just fkn works.
I think we are real inflection point between an analog world and a digital world. I also think that the Bronze Age never went away and analog techniques will just develop into craftsmanship.
I haven't posted a link to these Autonetics parts in while so I will do so again.
What I love about Autonetics is that they were solving the inertial guidance problem in whatever computing form they could get a hold of.
I'm working on a proper website but the images linked here represent the inertial guidance problem being solved with discrete components arranged in circuits and later those circuits integrated in silicon.
https://en.wikipedia.org/wiki/Autonetics
https://www.icloud.com/sharedalbum/#B0YG4TcsmGWIVSf
Some Nixdorf 820 photos
https://www.computerhistory.org/revolution/memory-storage/8/...
It's not just the natural order of things that such knowledge passes out of this world. It's one of many collective and affirmative decisions by people born before 1980 to take as much with them as possible when they go. I would be skeptical if it were just one thing, but after affordable college, affordable housing, affordable healthcare, affordable retirement, and a viable biosphere, it's become a pattern. That you seem to understand this on some level and don't care to try to mke a difference is even worse.
30 years is plenty of time to learn the everything from the browser down to the CPU, or the HTTP framework down, or the database down, and probably enough to learn two such slices. Part of why it seems impossible now is just because most developers not only don't have the historical context, they haven't yet had any time.
i think whats required to move forward isn't just some dusty studying of these physics, but a reawakening to an understanding that these are just systems with tradeoffs, and we can choose to make other systems just as easily.
thats the only way the sandpile collapses and we get .. for example .. secure operating systems and network protocols. or systems that were designed to exploit the massive amount of concurrency available in modern machines.
This is simply not my experience of young people or of being young. There's a reason why radical movements are led and populated largely by the young—to the extent they err, it's on the side of "let's tear it all down and start over".
Trying to learn all this in 1/10th of the time to start your career is what requires perseverance.
For example: You and I HAD to be willing to go through that, and we were fascinated by it. There are still plenty of great FE and BE young devs who see something and instantly ask "How does this work" and keep going down. The difference is, most people in employment don't HAVE to do this like all people in our early career did. If those shortcuts to getting paid existed in our early careers, many of us would have taken the money and not cared. It's not an experience or a "You had to be there" thing, it's simple curiosity and motivation that sets people apart in this regard.
Of course, it's bad if the western world fully outsources certain components of the stack.
But not everybody needs to understand the full stack. Maybe you don't know how the silicon crystals are made and that is okay.
Some people like to say this is a perennial problem that every generation bemoans. I get why that might seem intuitive, and for the longest time I've hoped they're right and I'm merely getting old.
Sadly they are wrong. Long term historical patterns of empire collapse follow this over-reach and under-education cycle that leads to a catastrophic rebuild capability gap that is triggered at some point.
Thomas Thwaite's "The toaster project" [0] is a wonderful commentary on this. I cited it in a talk I did about education in world that will no longer pay for teaching kids how to use chips and breadboards in skinflint universities that prefer to teach them Microsoft Excel or use some crappy simulators of everything [1].
When it comes to cybersecurity we can only obtain defence in depth if we have knowledge in depth, and there are ever fewer of us around who understand how computers actually work. It's more than a little unnerving to meet CISOs and folks with high-flying titles who have no idea why using a phone app to control a Fortune 500 company infra might not be a good idea.
I've been trying to write a blog post about this, and how to address the problem specifically for front end development, where it's absolutely ludicrous. It's a very serious problem, especially when you think about security.
You have all these people clamoring to get cloud certifications so they can work in IT but its so complex. A certification isn't enough. There just aren't enough people with enough knowledge and practical experience to fill all the needs that companies have.
If by "understand" you mean "have a passing familiarity with" then.. sure, but if you mean "my understanding is reflective of reality" then there's absolutely no way.
We, people who really understand the technology that the world runs on, are a slowly dying breed.
There are more people doing "low level" programming right now than were doing it a few decades ago. The only knowledge that is fading is knowledge that's applicable to legacy and/or retro systems.
I am someone who grew up with the technology, as the levels of abstractions were being added. I am now benefiting from all those accumulated decades of knowledge.
As the IT / development world was changing, I had enormous privilege and comfort to learn the things at the pace they were happening. Being able to assimilate changes over long decades. Be a witness to the problems and logic behind all those new solutions. Understand how we come to have JavaScript and the browser mess we are in and so many other curious features of todays digital world.
I understand pretty much all of the layers of the computing from how CPUs achieve some of the things they are doing to bus protocols, to instructions, physical memory, low level OS internals, high level OS internals, virtual memory, userspace platform communication with OS, programming language runtimes and linking, shared libraries, IPC, networking, virtualization, etc.
The issue, as with any automation, is that new players on the scene (younger devs, devops, etc.) simply have no chance to learn the same things and go trough the same path.
For them, spending a decade working with a low level programming language before you jump into high level programming language is simply not an option.
We, people who really understand the technology that the world runs on, are a slowly dying breed. We are still here as tech leads, managers, directors, business owners. But there will be a point in time when we will go on retirement and there will be only precious few people who had perseverance to really understand all those things.