This brings back memories.... At a previous employer we had a product of ours that would just randomly reboot. Standard Windows on an industrial PC, nothing you’d think would be that esoteric. It’d look like a sudden power failure. No crashes, nothing in the logs to indicate anything at all. Similar hardware platform to some other products we had where this issue never arose.

It literally took months to find the cause, while in the meantime the people that didn’t want us to succeed (think big-company politics) were scoring points against us. We did everything we could think of, fixed a lot of bugs, tried stress-testing the system, things like that. Nothing worked. We were met with skepticism when we started saying it was the hardware. They said it can’t be the hardware....

Except this time it was — and we found out almost purely by good luck. One of the hardware guys happened to have an oscilloscope hooked up to a system when the reboot happened. The reboot was preceded by ‘something strange’ on the scope (I’m likely missing stuff here since this is memory from a few years back).

It turned out the motherboard had a bug which would manifest when the system went into a certain lower power state. That’s why stress-testing never caused it. In fact, the testing we did was actively preventing us from catching it!!

Lesson well and truly learned.

On a related note - is it just me or did Intel have a lot of trouble with their power-management implementation a few years back? I’ve since worked on another product that would randomly hard-freeze and sure enough after checking the Intel errata doc for the CPU/SoC in question (an Atom something-or-other), sure enough there were issues with S-states that meant we needed to limit them (to S1 I believe) in the BIOS, which did indeed fix the freeze issue.

To anyone that wants to avoid the long road here, realise that modern Windows, Linux and macOS systems never freeze or kernel panic without a special reason. Part of what I had to unlearn many years ago was the notion that on Windows, freezes and blue-screens were things that ‘just happened’ in the course of otherwise regular operation.

Low-level can get you, even if you aren’t doing low-level work.

Problems like that are fairly common when you're developing hardware products. Even if you're using products / modules someone else made.

> On a related note - is it just me or did Intel have a lot of trouble with their power-management implementation a few years back?

Everyone seems to have troubles with their power management stuff! Power state transitions are tricky in hardware, firmware and software. There are a ton of corner cases and assumptions. For example, you have to be careful (in software!) that you don't cause voltage dips by switching on chips and their peripherals too quickly. See [0].

CPUs, chipsets and peripherals have pages and pages of errata that BIOS, microcode/firmware updates and operating systems work around.

[0]: https://en.wikipedia.org/wiki/Inrush_current

All of this is true, and you haven't even started getting into prototype hardware where a minor coding error could cause your hardware to literally catch fire. When you're playing with mechatronics, you often get into fun games of "was it mechanical, electrical or software that let the magic smoke out". :)

I heard the term “smoke test” came from the hardware expression “if there’s no smoke, it will probably work”

If you consider the various meaning of "smoke test" at https://en.wikipedia.org/wiki/Smoke_testing , I suspect you'll agree that the mechanical meaning is almost certainly older than the electrical or the software.

That's where you have some vessel that should be air-tight, and you blow some smoke into it to see if/where the smoke leaks. You could do that for plumbing, you could do that for musical instruments, you could do that for all sorts of things.

Maybe the electrical engineers came up with the term "smoke testing" all own their own, but I think it's a lot likelier that they heard it from some plumbers or something, liked it, and then it gained _additional_ cachet from the association with smoke-means-failure.

I don't think it's too much of a stretch to imagine EEs coming up with "smoke test" independently. I've never heard it actually used in the mechanical sense (although I think I was aware of the term) but I've heard many people talk about magic smoke and electrical smoke tests. And the usage isn't quite congruent - an electrical smoke test (as I understand the term, anyway) is very specifically the act of applying full voltage to a freshly assembled and untested piece of equipment and being ready to shut off the power if anything starts to smoke (or less dramatically, seeing if any circuit breakers trip.)

Indeed, a smoke test will demonstrate the integrity or not of one's plumbing stack. An exterminator used on to determine that rats were getting into our house through a vent pipe that had been removed without the remainder being capped.

Additionally, when bringing up new hardware, the knowledge you need in order to understand the state of a malfunctioning device and debug a crash is often stored in the head of the logic designer who wrote the HDL, not exactly something you can look up on StackOverflow.

It's the R&D equivalent of "contact your network administrator."

"But... but I AM the hardware vendor! sob"

It's always great when you need to figure out what a particular register means or the architecture of a particular error handling process or something like that, and you open the hardware spec and find "TBD" because it's all still unreleased...

Yeah... just need to have a ton of debug registers included. Especially if it's ASIC.

And yes, Stack Overflow is really not going to help you with pretty much any of this. When doing low level programming, better get comfortable with that there's often no one in the world who can answer your question. Find it out yourself.

Is low level programming as lucrative? I love it, but I don’t hear many people talking about the jobs

There isn't as much work in that area, but there aren't as many people that do it, either. It balances out to pay pretty well, at least in my neck of the woods.

Lucrative? Do you mean pay? Some low level jobs pay really bad, some very well.

So all I can say it might be, or it might not.