Timing, data cache, BTB, you name it: doing software AES exposes you to All The Side Channels, and AES is somewhat notoriously hard to implement safely in software compared to other software-profile ciphers. This is the big selling point for ChaPoly.