To be precise, it's impossible to measure one-way latency between a pair of nodes without external information. But if you have a mesh of nodes, the story is different [1]. If you have N nodes and hence N unknown clock offsets, and if you have ping data from N(N-1) pairs, you can do better.

[1] https://www.usenix.org/conference/nsdi18/presentation/geng

Well I think we need to distinguish between practical latency measurements on earth, where it's mostly a matter of rejecting dodgy data / outliers etc.., and detecting systematic latency bias that is theoretically impossible to detect. If you could get around this problem just by using a network of N nodes, then you could do stuff like measuring an absolute reference frame, thereby violating special relativity.

You should be able to tell when one side of the link degrades though, correct? You'd have known time deltas for the link each way (whether its accurate doesn't really matter) and when the round trip time changes you should be able to tell which side that occurred on. (I'm interested in this problem because my cable internet reliably (!) degrades during the workday)

The problem is physics can't even distinguish whether light travels instantaneously in direction x, but at c/2 going in the opposite direction. There is no known way to synchronize clocks or come up with a clever apparatus to prove that light moves at the same speed in every direction.

You need to make some strong assumptions in order to try and measure asymmetrical latency on a network. I believe the method you are proposing is to measure RTT when RTT differs depending on the starting node. The difficult problem is determining how much of your RTT time comes from node A to node B as opposed to node B to node A. I believe you need to at least have some sort of oracle on both sides which can synchronize time without using the network path you are trying to measure (I think you can do this if you have node C which both A and B have a fast connection to relative to each other, this implies that the delay between A and B is significantly longer than the speed of light delay).