ZSH only gets slow because of the customizations, which would be true in Bash as well. Both shells are relatively comparable, though there are still a few things you can do in ZSH that you can't do in Bash. If you keep your customizations relatively light-weight and don't iterate over too many files on each login session, you'll probably be OK.
I wrote a framework to help deal with the shell a little bit after getting fed up with OMZ's slowness: https://github.com/tubbo/homer
I wrote a framework to help deal with the shell a little bit after getting fed up with OMZ's slowness: https://github.com/tubbo/homer