If a flagship device came with glaring flaws - even in functionality I rarely use - no way I'm giving them the money unless it's really revolutionary in some other aspect I care about.
So working out initial problems on a budget phone makes sense.
Nexus phones also weren't really successful and weren't intended to be. They were intended to be development devices and for Google to show the direction that they wanted Android to go in. They also didn't build any of them in-house, they were made by HTC, Samsung, LG, Motorola, Huawei, and Asus.
Google's real entry into the smartphone market was with the Pixel which was a direct competitor to the iPhone.
- Nexus 5 was low cost and amazing for the price. I was able to forgive every issue it had. And another international friend also asked me to buy it & ship it to her.
- The 5X was not low cost, but then it could also be bought cheaply through the Google Fi "loophole." I know at least one other person IRL who bought it through the loophole.
For long, Google has mostly sold nexuses in order to have demonstration devices of what Android could/should be.
Also, they needed dev devices for their engineers working on the next Android version anyway so it made sense to go the extra mile and sell a couple of them as well.
I don't think any of them has ever been low budget (below 200$); some have been middle budgets and some other high. The Nexus 4 has been introduced at 300$ IIRC but it was not the rule by any length.
Andy Rubin thinking he could skip this process and just jump in at the top of the market was pure hubris, and as much as I was hoping to see them succeed, it's a little bit reassuring that they failed. Success takes hard work, and essential skipped a lot of that.