Open source router firmware: a) is derived from well-established Linux/BSD networking stacks, b) has a smaller installed user base, so is a less-attractive target, and correspondingly c) has a lower value of exploit, which is more likely to be CVE'ed for reputation than sold for cash.
An audit would be interesting and welcome, but I have more ambient confidence in open source router firmware than vendor firmware. There's nothing unusual about the hardware in these things that the vendor has special knowledge of.
Also, most alternative firmwares have better performance, a better UI, are more frequently updated if bugs are found, and have more features than most stock firmware.
I've used OpenWRT, DD-WRT, and Tomato on inexpensive consumer routers in the past. Never regretted any of them. I purchase home router hardware specifically by the compatibility lists.