I'm probably biased, but Solaris' role-based access control system has been very successful among its particular customer base:

https://docs.oracle.com/cd/E23824_01/html/821-1456/rbac-1.ht...

It wasn't necessary to throw away POSIX history or concerns either.

Try the first two to be certified to high-assurance security:

http://www.cse.psu.edu/~trj1/cse443-s12/docs/ch6.pdf

The MLS model was too difficult to adapt to commercial use. Biba was good for stopping malware from overwriting files. They still preferred something more flexible. SCC then invented type enforcement in another high-assurance system:

https://web.archive.org/web/20160311233659/http://www.cyberd...

Flask architecture was combining that tech with a microkernel. SCC, acquired by McAfee, added type enforcement to a BSD OS for their Sidewinder firewall. The next work by Mitre was proof-of-concept for OSS by adding it to Linux. That and a pile of incremental additions is called SELinux. I'm sure you'll find the LOCK design a lot cleaner as it was originally intended. ;)

Also worth noting are the KeyKOS system (esp with KeySAFE), the capability-security machines, and one language-based mechanism:

http://www.cis.upenn.edu/~KeyKOS/

http://www.cs.washington.edu/homes/levy/capabook/index.html

http://www4.cs.fau.de/Projects/JX/

These collectively should keep you busy for a while. They're the kind of thing worth imitating or building on.

Aside from well known options like Plan 9 and QNX:

- https://pdos.csail.mit.edu/archive/exo/

- https://atheos.syllable.org/

How about Magenta [0] by Google, on top of Little Kernel (LK), a neat and modern microkernel design? (It's part of Google's work-in-progress complete OS named Fuchsia, which appeared briefly a while ago on tech news sites.)

[0] https://fuchsia.googlesource.com/magenta

Microsoft's Midori project. It never saw the light of day, but there are some very interesting blog posts about it. The Redox project has some leanings in this direction.

Robigalia is interesting but not nearly ready: https://gitlab.com/robigalia (their website has a cert error right now. It seems like I've seen a lot of those these days)

It's a rust userland built upon SEL4. SEL4 is very simplified in order to meet their verification goals so robigalia has to implement some interesting resource sharing primitives on top of it to get things to work. It could be interesting.

Not sure if you're only considering Unix/BSD-ish OSes here or not. But there are perhaps hundreds of such projects.

http://tunes.org/cliki/operating_20systems.html

You may want to dig around that entire site to get an idea of what people have tried to do (and frequently failed).

Yes, Trusted UNIX where you have a very complex MAC framework