It's both actually. But that gives an apparent contradiction - how can both observers see that the other party has slower clocks? Who is 'right'?

The answer is that the 'moving' traveler needs to flip the ship around at some point and thrust back towards the starting point. Whoever turns back around will find that it's their clocks that passed more slowly.

And if the universe is spherical?

I think the question you are asking is what if the universe is toroidal. What if you traveled so far you came back around.

You come back around on a sphere too, e.g. traveling by the equator.

In that case you would be traveling a curved path so there would be acceleration even if your speed remained constant.

Isnt a torus curved? What's the argument?

If you use toroidal boundary conditions then you could remain in an inertial reference frame and return back to the same pont.

See https://en.m.wikipedia.org/wiki/Periodic_boundary_conditions

My instinct is that as you fly toward Earth at near the speed of light, earth-bound stopwatches will appear to be going nearly double their normal speed. As you fly away from Earth, the stopwatches will slow to a crawl approaching standstill.

This illustrates a fundamental misunderstanding of time dilation. Direction has no impact, and this example that is often used is terrible for illustrating what is being proposed re. time dilation, because it just has to do with the perception of time as it relates to light reflecting off of a clock within certain frames of reference.

I might be misunderstanding your comment, but you seem to be saying that the change in time is only actually thebvissible perception?

If so, that is not correct. Atomic clocks have measured actual, lasting time differences.

I was saying the opposite. The claim of time dilation has nothing to do with the visual perception of clocks, which is a common (flawed) example that is used. The visual perception of clocks differing by heading towards or away from a clock is just a phenomenon akin to the Doppler effect.

If you correct for this effect caused by changing distance between the parties, you will still observe a difference in passage of time in the two different inertial frames of reference (each observing the other). In both cases the party taking the measurement will observe that time is passing more slowly in the frame of reference that the measurer sees as moving with respect to the measurer's frame, regardless of whether that motion is toward or away from the measurer.

What happens if both parties are moving? They start back to back, fly a distance, both turn around and meet.

Each should see the other as dilated, no?

There is no difference between you moving with velocity X and the other party moving with velocity -X; or you moving with velocity 2X and the other party being stationary; or you being stationary and the other party moving with velocity -2X. There is no experiment you can perform that will tell the difference.

You missed the point that both parties accelerate, we are not talking about constant velocity here.