If something is sometimes a number, then it’s not a number. Address numbers in the US are just identifiers that use digits, and sometimes they are derived from some other factor that is a (or multiple!) number.