Because of how Git's current implementation of DVS works. There's nothing fundamental about it. Git already supports partial clones and on-demand checkouts in some ways, it's just not very ergonomic.

All that's really needed is a way to mark individual files as lazily fetched from a remote only when needed. LFS is a hacky substandard way to emulate that behaviour. It should be built in to Git.