Of course, an adversary can then figure out who you are connecting to based on the IP.
As an example, I tried a reverse DNS query for the IP address of matrixssl.org. All I got was a subdomain at gandi.net.
Reverse DNS was originally intended for troubleshooting. It is not required for websites (cf. email) and not everyone bothers to set it up. That is one group of websites where we have to do more work to get the names that are using the remote IP address and figure out which one the user asked for.
In fact, DNS is not required for a functional website. IP address of course works fine. That is another group of websites where we have to do additional work to figure out what is at the remote IP address. We do not know what the user sent in her HTTP headers.
By comparison, SNI makes the process of invading user privacy easy and reliable, less work. The user is required to send a name, and to send that name in the clear.
While the gp is wrong in saying that it's as simple as a reverse DNS lookup, his general idea is correct. It's trivial to crawl the internet to find all the ip -> domain mappings for all public domains. It's even easier if the attacker is an ISP because they can log DNS queries/responses.
If you connect to a website not behind a CDN, you probably don't have to use SNI, but, the website is revealed by doing a simple reverse DNS query. Privacy defeated.
Unencrypted SNI doesn't hurt privacy when compared to the status quo. Encrypted SNI will boost privacy. But until then, TLS is basically the best you can do for privacy, outside of using some more exotic service.