My browser also offers me to accept any self-signed certificate, I can investigate it and then I can accept it and it won't ever bother me again, until the certificate changes.

The problem is that this is a huge hassle for incidental visitors. Whereas SSH does not have incidental visitors. Same goes for email, if it's your own server, you know the cert to be the real one, and you can accept it, you're not bothered again.