Because the shell is used to do things that are easy in the shell. Examples: file manipulation, sorting, field based processing, pipes. Doing these things in a traditional language is more complex and not necessarily better.

Except the shell is bad for those things, except pipes I guess.

I just hate doing things with files in the shell, all those stupid escaping rules, and god forbid you have files with spaces or quotes in the names, or leading or trailing spaces.

It's just spending time learning hacks to get around shell limitations instead of actually getting things done, and at the end it's an ugly mess.

Or maybe I'm just bad at it, but I don't think it's only that.

It’s not more complex in Python if you use the sh package. All the power of everything you need with a real language when you need to do more.

Packaging in Python is a bit more of a pain.

That's the other part of the problem. The shell is available everywhere. Python needs to be installed. So, if you need shell-related functionality, it is the easiest route to use.

I have not encountered a system yet where Python isn’t installed out of the box.

Windows, possibly. And as for the *nixes, including Macs, what version is installed? The way Python 2 eol was extended and extended and extended just hurt the whole ecosystem.

Windows doesn't have bash preinstalled so I'm not sure why you're bringing that up. On all OSes I've used in the past 3-4 years all of them have been Python3 by default. The specific version is largely irrelevant as I could ask you the same question about which version of bash is running on a given OS (e.g. it's not available on Mac out of the box as the default shell is zsh due to GPLv3).

  > Windows doesn't have bash preinstalled

I meant Python preinstalled.

  > On all OSes I've used in the past 3-4 years all of them have been Python3 by default.

Windows comes with Python preinstalled now?