There is work going on about external object database support that could help in the long run:
https://github.com/git-lfs/git-lfs/issues/1702
(I am working on this for GitLab.)
I'd assume there's licensing issues before anything else: git-lfs is MIT licensed, git GPLv2.
The release notes mentioned protocol improvements with git-filter that can dramatically speed up git-LFS (the large file storage plugin).
Does anyone know if there are any plans to make git-LFS part of the base instead of an add on?