Swap has nothing to do with it. Getting malloc to fail is easy:

    malloc(-1);

With address space fragmentation, the requested size can be a lot smaller and still fail. With overcommit, you can allocate a ton of memory without using up any RAM.