Yeah, you're right. It's not an optical illusion. I think I'm wrong about the infinite acceleration part though. We never observe the person hitting the event horizon so time never actually freezes. It just gets really really slow and approaches the point of time freezing.

The person falling into the black hole will have time in the outside universe accelerate relative to him. It will get faster and faster and when he hits the event horizon we don't know what happens as that's the singularity where the math no longer works. We can speculate that time actually freezes but that leads to paradoxes.