Technically the allocation succeeded. That's why you have a pointer and not NULL. So the answer is the classic one: "you check if the pointer is NULL or not".

This entire thing is a wrong question to be asking, anyway.

Even if you are sure right after malloc that "you have the memory 100% available to you", who's not to say that, immediately one nanosecond afterwards, some other process comes in and, well, asks for more memory? The memory you were so sure you had for yourself right after malloc may no longer be there (either swapped out or even discarded if possible), and you will be killed for trying to access it again. Oops.

At no point you can say "I have this memory and it's mine and just mine!" unless you are root and mlock (or equivalents). And I hope you don't do that, anyway. Just let the OS do what it is designed to do.

There's gotta be some way to programmatically determine it, right? It may not be portable. It may require some system calls or something, but there's gotta be a way, right?

I guess you could mlock() your memory and see if that succeeds or fails. When overcommit is enabled no memory is really safe.

Maybe a system call that checks whether writing to a page would result in a fault? What if we had memory file descriptors for pages? We'd be able to use them with epoll and io_uring to asynchronously check whether it's safe to write to the pages they represent.

I'm not an expert enough to tell you; I just read the article and decided that it was too long so summarized for others. There's some discussion upthread about catching SIGSEGV and other methods, FWIW.

The question is what problem are you actually trying to solve.

You can get away with just about anything if you do it in a child worker process which can safely crash.