Prezto is not terribly fast either, in my experience. I went from oh-my-zsh to prezto, and from there to a simple config, always just to reduce bloat.