People with Shenzhen hukou can visit Hong Kong without a special pass (or to say, their hukou is a special pass). You also don't need one if you are transiting through Hong Kong and take a layover for a few days (that's how my wife did it on our trip back from Bali).

There are also plenty of Chinese tourists in Hong Kong, not just elites but a lot of middle class.