BoM cost is not the only cost: the larger your processor, the more complex things like power sequencing, routing to DRAM, boot sequence and bringup, etc. are going to be. Those affect the overall design time and cost even if you're only building one item.

I spent a couple of years building a system to run in the "Sierra Wireless OpenAT" RTOS environment (like those ubiquitous SIM900 modules, but you could add your own code to them). It was pretty terrible and I'm glad they now offer modules that run Linux.