ECC is pretty closely related to the study of prime numbers. It might not be built directly on the difficulty of factoring, but the theory of how to construct curves, how to use them, what's expected to be secure etc goes pretty deep.