OS X doesn't provide kernel features directly equivalent to Linux's CLONE_* and cgroups. But it does provide Seatbelt, a sandboxing policy layer, which you could use with effort to implement something similar in end result. This is what both iOS apps and desktop App Store apps use for security isolation.

Alternatively, you could ship a kernel module that added these features (although that might be ridiculously complicated).

Alex (aka vito) likes implementing things from scratch on weekends. So if you surprised him with a github issue he might just take you up on that challenge.