Look at it this way, let’s say you’re at the event site and seeing it happen right in front of you.

Suddenly, some travelers arrive and say they also came to witness the event, but they came from light years away.

How? The light of the event would have only traveled for a few light minutes, and no other type of information would travel faster than that. There’s no way someone light years away would know about it yet.

The only explanation, assuming they didn’t perfectly predict the event, is that they time traveled. From your perspective they covered a distance of light years in the span of time between the event’s start and you first meeting them.

Sure. But assuming time is traveling the same for you, their origin point, and the photons going between the two, they won't suddenly appear when the event has just begun, even if they travel instantaneously. Because, as you say, the point at which they first saw the event had yet to receive the light image of the event.

And if they do seem to suddenly appear much faster than you would believe possible, you suddenly know that you're at the bottom of a time-dilation gravity well (or the equivalent). Though you probably could have figured this out beforehand from the red-shift of the light traveling between the two points. Or you know that they observed the event from a closer point of view, instantaneously traveled to a most distant location to change clothes, and then traveled again to your location. Time travel is only one of at least three possibilities.

If you are in a time dilation gravity well, the mere act of outsiders traveling into it might be time travel itself, as they are traversing the well from the outside (what you would think of as the future) to reach a point inside where time is lagging behind.

Time may be lagging behind, but you aren't traveling backward in it. Merely transitioning to a slower tick-tock cycle.

So in that case traveling into the heart of the well would feel like you are fast forwarding the events of the well until you gradually synchronize with the tick cycle and things move at normal speeds? In which case you could never reach the exact moment of the event you set out to see.