When I was doing mail delivery as a summer job, there always were experienced delivery pros available to help decode addresses (a win-win situation: the guys approaching retirement had less mail to deliver, so they had lighter bags, but they could only leave when all others were done sorting their mail). During the sorting job in the post office, you would find postcards in your lot that only mentioned the colloquial name of a bar, the colloquial name of a regular there and a city name. When you asked, they would immediately tell you "that's x street, number 45". They also often could give you a house number, given a surname and a street.

At first, that seemed surprising, but even after a few weeks, you start recognizing surnames and linking them to numbers. It helps that mail is not uniformly distributed over all addresses.