That's actually what I do, and it works well for me. I switched back to gentoo when I decided to do that, precisely when I realized that trusting that binaries match their source code was unjustified (I had already a decade of using gentoo, so it was not a problem for me).
I also switched back to chromium, for that reason. Firefox is great, but it won't allow me load an extension from the FS permanently. All the extensions I use nowadays are loaded from sources (unpacked extensions, as chromium is calling them), after an inspection from me.
Of course, I haven't read the code of _all_ the programs I'm running on my system, so it's not perfect security. I'm still confident this is a better level of trust than running binaries.
A side effect of that is that since I actually _do_ read a lot of code from the programs I use, I learn a lot, and it often happens that I change code of programs I'm running to fit my need (the portage system of gentoo make it easy to write your own ebuilds and integrate your changes in your package manager).
Doing so require specific hardware, though, as if you're not careful on what hardware you take, you'll probably need binary blobs to use it.
There is still a security problem I need to solve. Some programs (well, chromium, mainly) easily take 6 to 8 hours to compile. So I usually just lock the version to a stable one and update every month. This may be a problem if a security patch is released.
Building from source is a tricky one and actually I'm not sure that everyone should compile from source (although it is definitely good to retain that ability, and to have, for example, entire Linux distributons that do so during package installation.
Signing keys and binary signatures should achieve most of the result without requiring energy and CPU expenditure on recompiling every time. Especially in the presence of continuous integration, those resource costs can spiral, and that's energy that could be spent on other needy human endeavours.
Regarding enough eyeballs - yep, it's hard to tell at the moment. When version bumping a single dependency by a minor release versiom, often it's possible to do this manually by checking commit and change logs. But it's hard to scale.
The longer-term solution there is likely automation: we should encode as much of that manual diff-and-review process as we can into automated security scanners. With good enough static analysis, and languages that can unambiguously express code as syntax trees, we should be able to generate 'tree diffs' and look within those for resolved, unchanged, and introduced issues.
Lots to do and better times ahead :)
Imagine you knew a large corporation uses LibreOffice. I doubt you'd need a million-dollar black market 0day.
Imagine an average user tries to perform a backup reliably ('this looks like time machine only it breaks restoring between versions').
Imagine you buy new blueooth headphones and you can't use high-quality audio codecs out of the box but need to compile something called an audio server.
Hell, in 2020, you don't even know for sure which application draws your browser window on the screen. Is it your browser, or some other process imposing your browser?
These are so many distribution-wide or ecosystem-wide issue that I honestly don't see the progress at any acceptable speed.
For both open and closed source you can have security bugs and vulnerabilities, and depending on the attitude of the developers or the community behind it could be or not easier/faster to be solved on open source, but you can't bet on that.
But what is at the very least harder (or at least is in your hand how far you want to go to check about it) in one side is to include unnoticed backdoors, trojans and other gross things against your interests as user.
And yes, backdoors and intentional security problems can be introduced even with a duplicated line (https://nakedsecurity.sophos.com/2014/02/24/anatomy-of-a-got...). Is not a safe against all protection. But it lowers the bar.
Many have been focusing on reproducible builds lately too, so you can verify that a binary is from a particular version of the source code.
This is better than not having the code at all, however this is a false sense of security.
First of all, you should be compiling your own binary from the sources, otherwise you are blindly trusting that those binaries you download are built from the original source code, which may not be the case.
Second, open source security relies on enough eyeballs reading the code independently and spotting the security holes or anything malicious but you can't know how many people actually did. Some software isn't popular enough, some other software contains millions of lines of code.
The same process would have to happen for each patch and software update.
The same thing happens with closed source projects, however. Less popular software will have smaller staff and it's more likely to contain errors and security holes, especially if it's an one man project. More popular software will have more staff working on it but if the software is big and complex, most people working on the project have never read the entire code and there's more lines of code that may contain issues.
Software is a giant mess