I think this could be done, probably easier than doing the same for video production. An audio signal is as simple as just a list of values...
Basically, we would just have every "channel"/"track" from a digital audio workstation rendered into a file. For very similar tracks only the differences need to be stored. Then we would also need one "master" channel which stores mastering information.
Now such a format would pose a couple of challenges. First of all, size. Lossless audio in flac takes quite a bit of space already, an entire album is usually around 100mb. This would use even more space. The next problem is that there would have to be a "standard" across digital audio workstations... how would tracks be classified? For example, the software Ableton Live allows one to group tracks and also group groups themselves. Effects can be added to groups but this people also use this feature just for organisational reasons. Other software might do this differently... Then the next thing which is probably the biggest challenge... the "player" would need to add the individual signals back together during playback and doing this in real time can cost performance.
All in all, I have been dreaming for years about managing songs and audio projects with something like git. It would open up a whole new world for collaboration and I think theoretically it would be possible.
Bouncing a full track to stereo is a lot like compiling software into bytecode/assembly. You get something usable and distributable to end users that you can reverse only partially. Part of that is intended. Just as with proprietary software, the exact combination of synthesizers, instruments, plugins and settings are closely guarded secrets of musicians, mixing and mastering studios.
And just like in software, some of those "shaders" aka plugins are actually proprietary themselves and can cost multiple hundred dollars each — so redistributing the "source code" can be not just difficult but actually illegal or impossible.
As you see, the incentives generally aren't aligned towards this form of distribution.
Just like in software though, I think there's a set of people interested in actually distributing their source tracks, made with open source software — and IIRC, that's exactly what's happening here as the base of these source separation efforts!