I believe this is because the yaml is in fact the instructions for what to include in the binary. It wouldn't be feasible for the firmware to include all possible device and peripheral code and enable parts at run time.

I think you can see the esphome intermediate code generation in the file tree during compilation and see how the yaml sections map to blocks of C/C++ code being built.