Great news - what you're wishing for mostly already exists[1]. The relatively new sysupgrade server and attended-sysupgrade clients automate the process of creating custom images matching already installed packages. After the breaking network config change a few years ago, it's now pretty safe to keep config files and ssh keys while flashing an upgrade image. The end result now is that I can seamlessly and painlessly update my OpenWRT boxes with just a few clicks or commands, despite loads of installed packages and config files.
[1] https://sysupgrade.openwrt.org/
I often delay updating my router because it takes ~20 minutes of work to reinstall packages, manually merge config files, and make SSH stop complaining about the host key.
There is an incentive to use as few features as possible, because every divergence from the base config is a perpetual maintenance burden.