Yeah, in the original gpt-3 paper one of the more interesting bits was that it made similar off by one errors a human would make when doing arithmetic (and they controlled for memorized test data).