Apple's Core Storage does have some COW properties, so they could probably do something like linux LVM thin provisioning style snapshots. They could maybe add a tree for checksums, which would be a good idea for the journal at least but seeing as this hasn't been a priority for even just the journal, the most unencumbered and recent add-on, it seems obviously not a priority. The COW stuff was necessary to automagically migrate data from the SSD to the HDD in their so called "fusion drive", so maybe they leverage that for an updated version of Time Machine and could also do it for atomic online OS/software update with an optional later reboot rather than a mandatory one.
One rather cool thing it does is on-the-fly encryption and decryption of an LV. You can even reboot in the middle of either conversion and it resumes where it left off. And those encrypted volumes can be resized (that's not well documented but is supported and is even used by their own Boot Camp Assistant).
So about the limit of the new fs is an LVM like thing under journaled-HFS+/X.
And even if they did build a new FS, there are some assumptions like case insensitivity that likely couldn't be abandoned without potentially breaking backwards compatibility.