Nearly everything I learned about embedded systems programming came from either Embedded Systems Magazine articles(https://www.embedded.com/embedded-systems-design-magazine-ar...) or just being a paranoid SoB.

Writing reliable systems means spending a lot of time thinking about what and how failures can occur, minimizing them and dealing with the cases you can't get rid of. Simple is nearly always better.

It also helped to be an EE by education and not a CS major. CS majors can get blinded by their abstractions and forget that they're working on a real piece of HW with complex limitations which require design tradeoffs.