This is a fair point, but I simply want to point out that was half the reason I suggested it. Learning how to fail :)
I very much understand the mental aspect here though, so that's a tough one and my advice is not for everyone. I will say that at some point in your career you will fail though - or you aren't pushing yourself very hard. I believe this was one of the harder things I had to "train" myself to deal with; the mental load of failing at something and how to recover.
If you're not in a mental space to make that experience a positive for you? Absolutely skip it.
It's really worth it, IMO. There's really a HUGE element of luck: from the personality and opinions of those interviewing you to the questions they happen to choose that day. Seriously - internalize that and just try it out!
The cost of the flight & hotel is pretty trivial in the grand scheme of things. By that I mean relative to stuff like going to a place you want to see, the value of spending a vacation day, etc...
I do agree that the meeting smart people is fun... but the full on crash-and-burn on an algorithm + whiteboard interview would really have a massive negative impact on my confidence & self-esteem. Not worth it.