And I believe this is because they want anything that might be useful but competitive with the features they provide their users to be hamstrung by the same design contraints of iCloud and APFS.
My guess is that the number of users that use remote file storage providers is just much larger than that of users using actual (non-SMB) network or local (non-(ex)FAT) file systems, so they created a kext replacement for that use case first.
If anything, the File Provider API levels the playing field between iCloud and third-party remote storage providers, since the neat APFS tricks they can use to implement it would otherwise be unavailable to third party developers. For example, I find Google Drive to be much more well-behaved on File Provider than it ever was when using FUSE (I blame it for some hangs or kernel panics) or SMB (which was more stable, but did not support Spotlight at all).
For all the other use cases, I'm hopeful that we'll be seeing a real macOS FUSE sooner rather than later.
iCloud Drive is a File Provider. Third-party services that want to do the same kind of file syncing as iCloud Drive, and thus compete it with it, can. And there are several highly-popular competitors that have recently moved to File Provider, like Dropbox and Google Drive. But when they did so they had to lose some functionality, like the ability to store data on an external drive [1]. And for the less popular use case of third-party filesystems that aren't just for syncing, File Provider is completely inapplicable.
The box covers most of the use cases that most users are using third-party filesystems for. But it's shrink-wrapped around those use cases – no room to invent new ones.
[1] https://talk.tidbits.com/t/dropbox-drops-support-for-storing...