One benefit I can think of is that the volume of addresses that the average person reads tends to decrease with distance from the person's home. This means that in a left-to-right, top-to-bottom language, you can start reading from the beginning of the address, and as it gets more general, eventually you hit a point where it becomes identical to your address for the rest of it, so you can stop reading. For example, when reading an address in the city I live in, I can read the number and the street, and then once I see the city name, I can stop, since I know everything I need to. This would be a bit more difficult if you had to start from the bottom right and work your way backwards, so I'm guessing that the ordering was chosen to maximize efficiency for those common, local cases.

Exactly, it's the same reason most date formats don't start with the year, even though it is the most significant of the numbers. If I write 12-8-2019, the 12 is the most important number for me, in speech I will usually refer to the date as "the 12th". In the same way, if I'm telling my address to someone, I'll just say the street address, since we can assume I live in the country we're in right now (which of course doesn't work on the internet, hence we specify additional information at the end).

Yes, and at least historically, you can also write only the short part of the addresses that differs from yours. Just put the street address if sending it in the same town, and let the postal service take care of the rest. Today sorting centres are more centralised and that's not a good idea. But you don't include a line for the country if you're sending a letter domestically.