OS X doesn't provide kernel features directly equivalent to Linux's CLONE_* and cgroups. But it does provide Seatbelt, a sandboxing policy layer, which you could use with effort to implement something similar in end result. This is what both iOS apps and desktop App Store apps use for security isolation.

Alternatively, you could ship a kernel module that added these features (although that might be ridiculously complicated).