Check your Zsh theme. Zsh has no built-in support for git status, so themes that show git status are implementing this by themselves. One way to make this slow is to check git status synchronous, however there are ways to do this asynchronous that should be instantaneous (at the expense of showing git status later).

Btw, Bash is the same. So you just said that someone that implemented your Bash theme did better than in your Zsh theme.