Most people running macOS (as vemv at least is) run a lot of random binaries from various types of strangers – mostly proprietary applications, although there's also things like Homebrew cached binaries, 'curl | sh' installers, etc. Now, in this case downloading a binary isn't necessary, since there's the option of building it yourself. That's great, and it's definitely (somewhat) safer to build it yourself... but that doesn't make the binary less trustworthy than if that weren't an option. If building sounds like too much effort, you shouldn't automatically give up on using the program, but just fall back to your personal baseline policy for running binaries. And for a lot of people, for better or worse, I think that policy is fairly trusting.
That's wildly different from the typical CI-built binary.
Seems like there's a simple solution to this dilemma. Assuming build time is the single determining factor:
If the value of two hours' build time overnight is less than the value of a privacy-enabled build of Chrome, then build Ungoogled Chromium overnight.
Else, get Firefox.