With overcommit you will only get NULL from malloc() when you run out of the address space (or from the subset of address space that is used by malloc). When you overrun the address space ulimit, you are killed by some signal.

That's not right: http://pastebin.com/aA7dasDx

  jes@themisto:~$ cc -o alloc alloc.c
  jes@themisto:~$ ulimit -v 100000
  jes@themisto:~$ ./alloc
  malloc returned NULL after allocated 94227K

EDIT: Ubuntu 15.04, x86_64, Linux 3.19

however, ulimit will only affect your process; if other processes consume enough RAM, you can still run out of pages while under the influence of ulimit.

OK, mmap() will fail and thus malloc() will return NULL. The point is that you cannot rely on that. For example try what will happen with recursive functions that call alloca(1024).

Sorry how is this relevant to heap allocation? alloca just allocates memory off the stack - it's literally just bumping the stack register by the number of bytes you requested (and hence it's freed when the stack pointer is reset on returning from the function call). Alloca should fail when your current thread runs out of stack space - which will generally be only a few megabytes. The total stack for a new thread is allocated from the heap at its creation, not during its execution and certainly not as part of a call to alloca.

"malloc() won't fail"

"oh yes it does"

"oh no it doesn't"

demo of malloc failing

"alloca() won't fail!"

Stop moving the goalposts. Besides, alloca() tends to crash your program a lot sooner as stack allocation is usually very limited, especially in a shared-memory multithreading context. Hardly anyone uses alloca().