It won't be catastrophic but it could affect them. Most pacemakers have a mode that if you put a magnet on them they revert to pacing at 60 beats per minute (VVI mode) like a metronome rather than more advanced adaptive modes (like DDD mode). That's how if a doctor suspects it's malfunctioning due to bad sensing they can get it into a basic mode. I've done it in the ER when a pacer was freaking out and going way too fast due to bad sensing.