Eh... it's a lot more than that. It's the specific packages themselves, and how those packages are built, and how the OS is structured, which versions of libraries are used, how certain common tools are named (think python vs python3). The philosophy of different distros can be radically different. Startup environments can be completely different. Defaults and config in different places. Packages are patched differently. Libraries named differently (think dynamic linking to specific library versions). Init vs. systemd. Wayland vs. XWindows. Some distros even eliminate the distinction between /bin and /sbin. If RHEL, for example, adopted apt, a RHEL pkg likely won't work on Debian. That's why Snap, Flatpak and AppImage are what they are: completely self-contained or statically linked with no dependence on system libraries or image trees.

Actually, the packaging format and tooling is only a very small part of the time spent by the people contributing to the distribution. Go read debian-devel - only a very few people are involved in those things.

As far as I can tell, almost all the time is spent polishing and integrating packages and fixing temporary regressions, that sort of thing.

RedHat and Debian have different standards for various aspects of the package contents, so you can't just use a package from one and expect it to work on the other, even if the package format and install tool was the same.

Good. There are way too many distros anyway.