I found Linux From Scratch[1] invaluable when I built a system with it almost two decades ago (but I assume it's still awesome). The trick is to not use the automated build portions and read and follow their page on every step. It takes a while, but you come away with a much better understanding of how the system works.

1: http://www.linuxfromscratch.org/