If you are talking emacs, or the kernel, or gcc, there is probably enough interest there (and strong personalities in charge of the projects) to keep things safe.
I don't know where the cutoff is. At thousands of users you probably can't count on someone else having looked through it. At millions you can.
However you don't have even the possibility with closed-source products. That means you must instead rely on the process used to create the software. So something by MS might actually be pretty trustworthy from a "security issues not accidentally introduced" standpoint at this point after years of improvement on secure coding practices.
But they still operate by a profit motive, and there's still the possibility of deliberate introduction of security issues when it suits business purposes or for legal compliance.
Unfortunately there's no great way to tell that an OSS project from RANDOM_DEV is well-coded without looking at the code and it may be that a well-intentioned but junior dev introduces a relative swath of security bugs by accident.
I don't know where the cutoff is either. It may depend more on your threat model than anything else.
It's probably possible to verify closed-source binaries by decompilation and a lot of effort, if you wish.
But on the other hand between compilers/hardware/OS/etc. the part of the computer that is most susceptible to being attacked (the application software) is the part that's easiest to audit with open source.
If your chip's built-in RNG is cracked you can simply refuse to use it.
But if your libopenssl.so or CRYPTAPI.DLL is cracked you're probably screwed. It's at least somewhat straightforward to build your own libopenssl.so (or at the very least, verify that your distro's signed package RPM builds the same sha1sum from the known-good source tarball + patches). There's nothing you can do to independently verify the source used for CRYPTAPI.DLL, and even the binary of that DLL requires trusting Microsoft (and figuring out which Service Pack, KB patchset, etc. are installed).
However in the end you do have to trust people no matter what you do, so if you're willing to trust FooCorp's binaries then just go into decision with eyes open. Even most open source users are effectively no different in that they simply trust what Fedora or Debian are pushing out there.
But even one alert user is enough to catch a hacked .deb or .rpm, so you don't need most of the users to be paranoid. It's kind of a reverse herd immunity, if you will.