When synchronizing two nodes A and B, where there is a persistent difference in the travel times A->B and B->A, how do you achieve synchronization when knowing A->B->A or B->A->B?

you can't. You can only assume that they are equal and attempt to make them as equal as possible. (the same issue arises when measuring the speed of light: it's actually not possible to distinguish if the speed of light is different in one direction to another, we only know accurately the average of each direction)

For a first approximation, you can assume A->B and B->A travel times are equal.

And because optical links are used, the asymmetry is mainly due to the wavelength difference which is known.

Delay symmetry is a critical assumption in any two-way time transfer process. White Rabbit goes to extreme lengths to maintain that property.

This includes mandating use of cables that share a single optical fiber, with specific wavelength pairs and fiber types so you can calibrate for unavoidable differences in propagation time.

More info on their wiki:

https://ohwr.org/projects/white-rabbit/wiki/SFP

What happens when the roundtrip time isn't consistent?

The roundtrip time is never consistent. Light travels with different speed in fiber depending on the temperature. This is why you calibrate every second.