For me, it's not so much losing my train of thought but it drops me out of any sort of real conversational mode. At which point, I can feel like I'm giving up any sort of real engagement and just saying words.

(Though, in my experience, simultaneous translators I've worked with and listened to are pretty good. They seem to usually do a better than decent job even for speakers going along at a "normal" pace.)