One good reason for using Linux instead of any of the RTOS is better networking support. I've used VxWorks, RTEMS, Nucleus, FreeRTOS, eCos, etc and their networking stacks are all pretty slow and crappy.
I recently did a project with ucLinux on an LPC1788 (Cortex-M3). Getting the USB host stack is always a pain in the ass, but it was dirt simple on the Linux port.
Same for LCD framebuffer, GPIO, touchscreen (unique animals every time), I2C, SPI, etc etc etc. And I can pull anything I want off the kernel tree and not have to spend a week getting it working on another OS.
but that baggage also includes such juicy things as a hardware abstraction layer with plenty of drivers, a network stack with 6LoWPAN and a familiar programming interface.
Typically, the scale of embedded systems runs from very BoM cost sensitive (i.e. making millions of the things), to very cost insensitive (i.e. tens of high value items).
Microcontroller-based systems tend to be on the very-cost-sensitive end of the scale due to the fact that if cost isn't an issue, stick a i7 in there instead!
If you're cost sensitive regarding the hardware, then you want the most efficient software possible to reduce the hardware requirements, and hence would not want the baggage of a general-purpose OS for a build-time-configured, static system.
There are plenty of hardware abstraction layers in the embedded world... (e.g. CMSIS), and familiarity is just a case of what you're used to, that argument could quite easily be turned on its head with a different set of people.
IP stacks also exist and are well ported to many RTOSs, (e.g. LwIP).
For example, a (very) typical embedded system that would be familiar to a lot of embedded devs would be: - Cortex-Mx (CMSIS for HAL) - FreeRTOS (scheduler) - LwIP (IP Stack) - Eclipse IDE.
There are always exceptions to this, e.g human-resources and development time issues change the direction you head in.
I spent a couple of years building a system to run in the "Sierra Wireless OpenAT" RTOS environment (like those ubiquitous SIM900 modules, but you could add your own code to them). It was pretty terrible and I'm glad they now offer modules that run Linux.
I agree that nommu Linux is a bit to heavy for most cases as it will always require external RAM and at that point you might as well consider a cheap Cortex-A chip (well, they are not so easy to source/aren't documented as good as e.g. a stm32) but I'd be more than happy if something like RIOT or ChibiOS got more traction.
If its just for an IP stack, there's easier, more standard, less resource intensive ways of doing that than using a version of Linux with all its drawbacks.
I probably would have looked at lighter options like VxWorks or QNX or something but for our new products with a dual core A9 Linux suits us really well.
Anyone wanting a scheduler on a microcontroller based system would not want the baggage of a linux-level general purpose OS. You use a simple scheduler such as FreeRTOS/RTX/etc.
These days the only systems without full MMUs are microcontroller-based ones for which this is not appropriate. No embedded system is going to be dynamically-loading applications like this, its all bare-metal, all built into the one image.
No-MMU linux did have a use back when say the 68000 based lines were still in wide use, but these days all I can think of is for user-space linux where everything is simulated in a single process under a full general-purpose OS.