Yeah, letters are much more used in situations where they do not denote some sort of order. They do, at least in English, but it's a much weaker association. It seems on some level we intuit that letter orders are arbitrary, whereas in some sense integers are literally nothing but their order.