I recommend you try to do it -- it's a fun exercise, and will very quickly answer any remaining doubt why it can't be done manually.

We actually did that in university; the public key we used was something like 91 (not hard to factor). Wikipedia has an example with 3233, and that's already very cumbersome: https://en.wikipedia.org/wiki/RSA_(cryptosystem)#Example

I did that in university before you born

It can be done manually easily

I tried generating an Ethereum address by hand starting by creating a random private key. It quickly got away from me.

I wish there was some way to simplify the math to allow doing it by hand. It would be a nice way to check for correctness while computing it offline, and while ensuring dependencies hadn't been compromised to produce a fake public address.

I feel like making your own circuits that could do this would be a more feasible approach. Or at least a circuit that uses small purpose build ICs that are easy to verify.