I think fundamentally, there are only a few things it has to do: Allow applications to run (on the CPU/other hardware), Allow applications to communicate between themselves (establish communication standards), Allow access to disk, Manage the time given to each application well/fairly, Define permissions around resources (who can talk to whom and who can access what). Depending on who you ask, shells/desktop environments should be part of the OS too.
When I think of the minimum that can accomplish that, I think something like a simple communication standard (with authentication mechanisms) would be interesting (with possibility to support more specialized standards). A standard for defining app. share of resources (CPU/disk). Maybe hardware resources like disks, devices like cameras, etc.. could be treated as applications via a driver (or application <-> disk application <-> disk driver). Then instead of 'opening' files, you're just communicating your intentions with a disk driver, and you can do essentially anything. Maybe then instead of a Filesystem Hierarchy standard, there could simply be "OS (standard) applications" that for example list what users are in the OS, what applications are available, and so on (without a filesystem hierarchy at all, if you wish!). An FHS could be provided for legacy reasons.
I also think permissions should take a more fine grained approach than Unix/Linux does (closer to Android permission systems), I think by default applications should have minimum permissions and should be whitelisted as needed.
Unfortunately, today, UNIX is by and large all we have. Other things are either some kind of UNIX but with a twist, or very underdeveloped.
Looking into the future, I'm very enthusiastic about OS-as-a-library approach, but I don't think we have a solid contender there yet.