Two examples come to mind.

The first is process limits.

The second is exhausting kernel data structure space for things like page mappings. I've seen this recently in AIX when allocating lots of memory that alternates mprotect permissions.