As I understand it gravitational waves aren't slowed down by anything whereas electromagnetic waves interact with matter along the way, which takes some amount of time.

There isn't much matter to interact with, but not quite zero either. Something like 1e-23 of the distance had some amount of matter in it (mostly very diffuse gas clouds, nothing more).