Yes! Usually it's not a problem, but every now and then relativenumber really kills performance.

The last time I noticed this was with valgrind logs; and I have no idea what exactly is causing it (amount of lines doesn't seem to be it).

Thankfully it's perfectly usuable for day-to-day editing. Binding :set rn! to a key circumvents this problem well enough.