The purpose of the horse analogy wasn't so much to equate ChatGPT to a horse that just stomps, but the behavior of the humans around it who believe that it can count. This can be the naive trainer too, who doesn't recognize that he's subconsciously cuing the horse. He has a confirmation bias whereby he or she rejects any evidence refuting the hypothesis that the horse cannot count.

There is a difference between solving and problem and presenting a solution, and that difference doesn't hinge on whether the solution is correct.