From when I last looked, my impression was that the specific modular arithmetic you need for ed25519 is basically standard long arithmetic with a constant number of digits + a single CMOV, which is different from essentially arbitrary moduli that (reasonably compatible) RSA needs.