I read "come to harm" as "get harmed" instead of "do harm".

In simpler terms: what's good for some is bad for others.

Yes, the idiom "come to X" does mean "be acted on by X". Common usages "come to harm", "come to Jesus", "come to his/her/your/my senses" ( = rational thought).

It's especially confusing in this case, where the previous sentence refers to a place.

"I come to the beach to relax, but others come to party."

“Don't it always seem to go That you don't know what you've got Till it's gone They paved paradise And put up a parking lot”