RSA on pen and paper isn’t that bad compared to some other calculations people used to do by hand. By comparison in 1800 multiple people had already calculated the first 100 digits of pie. https://en.wikipedia.org/wiki/Chronology_of_computation_of_π

People can get shockingly good at that kind of thing with enough practice.