The reason the shell is used everywhere is because it's guaranteed to be installed (although DHCP used Bash explicitly), is much faster to start and run a simple script than Python, and its syntax is the command line that everybody should be familiar with.
It seems to me that a lot of shell scripts could be ported to other languages. Does DHCP on Linux need to use a shell script instead of python or something like that? The benefits of the shell grammar and semantics which are designed to make interactive use easy seem more like hindrances in a lot of those kinds of use cases. I assume it's largely done to make it easier for sysadmins to customise things. If I was a sysadmin I'd much rather learn python (and feel like I actually understood it) than the crazy byzantine grammar of bash. Maybe that's why I'm not a sysadmin.
This paper by Rob Pike might also be of interest: http://doc.cat-v.org/bell_labs/structural_regexps/se.pdf
>The current UNIX® text processing tools are weakened by the built-in concept of a line.