News for hackers. Stuff that matters.
Re:Tell HN: AWS appears to be down again by lordnacho on Wednesday December 15, 2021 @11:16AM (#29567966)
Surely we are way past the point where someone knows how the whole thing works, all the way down.

I doubt even a very skilled engineer would know how his own machine works all the way down. What I think happens mostly is the skilled dev can use his experience to know where to investigate and where to look for solutions.

The question is organisational. Might it be that certain orgs have gotten so convoluted that they cannot do this investigation on an org level? Essentially, letting the right people look in the right places, unhindered by politics, legitimate security concerns, and practicality?

You'd think there'd be a limit to scale at some point. A bit of redundancy makes sense. There's probably a lot of people with multicloud setups patting themselves on the back at the moment.

Re:Tell HN: AWS appears to be down again by gitfan86 on Wednesday December 15, 2021 @11:23AM (#29568046)
I do contracting dev work and my specialty is being able to drill down into any part of the engineering assets, ops, sec, dev. People think someone like me is slow and expensive until they have a problem that no one else wants to touch.
Re:Tell HN: AWS appears to be down again by whimsicalism on Wednesday December 15, 2021 @11:50AM (#29568422)
> Surely we are way past the point where someone knows how the whole thing works, all the way down.

So you are disagreeing with this statement and saying that, in fact, you are the person who knows how the whole thing works?

I knew tech work produced some large egos, but sheesh.

Re:Tell HN: AWS appears to be down again by rch on Wednesday December 15, 2021 @12:23PM (#29568904)
This type of person exists, and while rare, not as rare as some seem to assume.
Re:Tell HN: AWS appears to be down again by whimsicalism on Wednesday December 15, 2021 @12:31PM (#29569007)
There is nobody who exists who would be able to recreate a modern computer from scratch.
Re:Tell HN: AWS appears to be down again by 8note on Wednesday December 15, 2021 @01:22PM (#29569660)
No one person could recreate the pyramids from scratch either, and that's a pile of rocks
Re:Tell HN: AWS appears to be down again by whimsicalism on Wednesday December 15, 2021 @01:26PM (#29569712)
But they could teach others the principle and have them follow their directions to recreate it.

I'm saying no such person, even one who built up a team that they taught, exists for the modern computer.

Re:Tell HN: AWS appears to be down again by gitfan86 on Wednesday December 15, 2021 @12:40PM (#29569116)
When you say 'modern' do you mean with photolithography?
Re:Tell HN: AWS appears to be down again by whimsicalism on Wednesday December 15, 2021 @01:11PM (#29569512)
Yes, I do, keeping with the spirit of re-building the metaphorical "baths" of modern civilization.

But even if I didn't mean that, there still is nobody who could do it.

Re:Tell HN: AWS appears to be down again by sulam on Wednesday December 15, 2021 @11:48AM (#29568392)
Not at all diminishing what you do, but surely you have a limit past which you say "that's outside of my expertise, or what's reasonable for me to gain expertise given the scope of this issue"?

For instance, I manage a team that does "full stack" development, where full stack means I regularly interact with mechanical and manufacturing, operations, electrical engineers, battery and radio people, embedded developers, mobile, and most aspects of backend engineering. We had an issue where one of our chip suppliers changed their FW, didn't tell us, and we literally were taking apart units to get to the bottom of why units off the line weren't working properly. We go pretty deep. Still, at some point we throw our hands in the air and say "Hardware is hard, it's in the name."

Re:Tell HN: AWS appears to be down again by gitfan86 on Wednesday December 15, 2021 @12:00PM (#29568591)
This was meant to be in the context of hosting software services on AWS. Certainly there is a limit. If a MBP get a crack in the case, I'm not going to figure out how to machine a piece of aluminum into a new case, I'll replace the laptop.
Re:Tell HN: AWS appears to be down again by opportune on Wednesday December 15, 2021 @12:08PM (#29568694)
You know how to debug kernel issues? You know how AWS virtualization works and how to diagnose a problem with the AWS networking stack?
Re:Tell HN: AWS appears to be down again by gitfan86 on Wednesday December 15, 2021 @12:28PM (#29568964)
You know how to debug kernel issues? Yes

You know how AWS virtualization works and how to diagnose a problem with the AWS networking stack? Yes, and yes assuming that everything AWS is responsible for is operating within spec. Obviously, I don't have access to their switches, and cannot see anything at layer 1 or 2.

Re:Tell HN: AWS appears to be down again by dijit on Wednesday December 15, 2021 @12:26PM (#29568939)
These are not as hard things as you make them sound. They are the things traditional sysadmins spent time understanding.

But I am also incredulous by the parent.

I wouldn’t be able to diagnose traces on a motherboard or a defective (but partially functioning) CPU.

I wouldn’t be able to diagnose irregular voltage conditions or drop offs.

The amount of stuff that I know I couldn’t diagnose is absurdly high, but the amount I don’t even know that I can’t diagnose is higher still.

And my job, like the parents, is to drill down and spend time in specifics.

Re:Tell HN: AWS appears to be down again by hiptobecubic on Wednesday December 15, 2021 @11:48AM (#29568385)
I also specialize in being great at everything.
Re:Tell HN: AWS appears to be down again by swiftcoder on Wednesday December 15, 2021 @11:34AM (#29568198)
> I doubt even a very skilled engineer would know how his own machine works all the way down

Knowing how it works, and being able to build a new one, are also two very different problems. For example, there are plenty of Computer Science folks who learned how to design chips (layout the circuits, write the microcode, etc) - but you need a whole extra background in EE and Physics to be able to fab said chip...

Re:Tell HN: AWS appears to be down again by olooney on Wednesday December 15, 2021 @12:23PM (#29568896)
Many programmers complete some kind of nad2tetris[1] style course where they go from basic hardware primitives (the NAND gate) all the way up through a small von Neumann architecture computer that can be programmed with a simple homebrew machine code. Even if they don't, a good CS undergraduate program should cover a lot of it, and since most EEs can program at least a little they probably get a pretty good top-to-bottom understanding as well.

The problem is that this is really only possible with a toy model of a computer and very simple programs. Modern chips with their branch prediction and caching and threading and advanced vectorized operations and so on are vastly more complex. The 6502[2] was perhaps the last chip that one person could fully grok. Maybe a chip designer at Intel or AMD could understand the whole circuit in detail but no one else has the time - it would literally be a full time job. The same thing is true for operating systems - even if you're Raymond Chen, you can know a lot about Windows, but you can't know everything.

We learn just enough about the other parts of the system to convince ourselves that we understand the principles. We build the basic mental model we need to interact with other systems but all we can really do focus on our own specialized areas and hope that everyone else is doing their job. This works well enough until something like Spectre[3] or Meltdown[4] crops up and that's when we realize that we've been building castles in the sand.

[1]: https://www.nand2tetris.org/

[2]: https://en.wikipedia.org/wiki/MOS_Technology_6502

[3]: https://en.wikipedia.org/wiki/Spectre_(security_vulnerabilit...

[4]: https://en.wikipedia.org/wiki/Meltdown_(security_vulnerabili...

Re:Tell HN: AWS appears to be down again by FooHentai on Wednesday December 15, 2021 @12:27PM (#29568959)
I mean you can (and I have) etch your own circuit boards, but that's obviously not at the scale you need for anything other than primitive processing (70s 8-bit at MHz scale), and even then you're just looking at the next layer down (how to make your own chemical wash and get copper onto a board) as a barrier if we're truly talking about 'from scratch'.

We really depend on three things - knowledge (stored collectively and in various media eg books), materials (tools and manufactured precursor goods, available via active supply chains or existing stores), and most importantly having our basic needs met trivially so that all our time is not sucked up addressing them.

A scenario where someone has a 'wasteland' to pick over for their basic needs, knowledge and materials looks quite different to a return to primitive living where what nature provides is all their is to work with. 'if you want to bake an apple pie, first you must invent the universe' or however it goes...

Then of course there's the question of why someone would have any interest in obtaining computing power were either of those scenarios to occur. Much like the 'how do we warn future civilisations about our nuclear waste' problem perhaps it is acceptable to not bother, they'll figure it out again on their own eventually given enough time.

This stuff is fun to think about at 6am when hay fever is preventing my sleep :)

Re:Tell HN: AWS appears to be down again by SteveNuts on Wednesday December 15, 2021 @11:44AM (#29568342)
> There's probably a lot of people with multicloud setups patting themselves on the back at the moment.

And there's probably an equal number troubleshooting why it didn't failover the way it should, while their upper management starts questioning what they're paying for.

Re:Tell HN: AWS appears to be down again by com2kid on Wednesday December 15, 2021 @11:44AM (#29568343)
> Surely we are way past the point where someone knows how the whole thing works, all the way down.

I've met a few people who can rightfully lay claim, but yeah, an incredibly rare set of skills.

That said, there is a recent revival in building systems from the ground up. While you can't manufacturer your own transistors, it is quite possible to understand everything from simple logic gates to ALUs to older style CPUs and memory buses.

Re:Tell HN: AWS appears to be down again by cecilpl2 on Wednesday December 15, 2021 @12:23PM (#29568900)
I built a toy CPU in software once as an exercise. I started with "class Transistor" (wrapping an AND op) and "class Wire" (wrapping a boolean), and wired them together incrementally to make gates, flipflops, registers, etc.

I eventually got a fully-functioning 32-bit cpu with instruction pipelining, two levels of cache, DMA input/output, an asynchronous bus, a custom assembly language with an assembler written in python, and got the Game of Life running on it.

It ran about 2kHz with 8kb of memory or so.