The Raspberry Pi is based on an obsolete old graphics chip called Video Core IV with some ARM cores added as an afterthought. Of course it's cheap, it had a 10 year headstart and benefited from the economies of scale of the original product. And as you can figure out almost every SBC is based on old TV boxes (sometimes laptops like RK3399) because that way you can just slap the old SoC on a PCB and sell it without making a completely new SoC. The vast majority of phone SoC's simply do not have the interfaces (often just a single port for USB, network, and eMMC storage, display, perfectly designed for the phone in question) required by SBCs.

In short the production run is very small and that's why it's so expensive. ARM SBCs with custom SoCs cost around the same. Just take a look at linaro's ARM desktop [0] It's $1200 and doesn't even come with 8GB of RAM.

[0] https://www.96boards.org/product/developerbox/