I've been a designer in adult for 10+ years now and your correct about not being able to use your work portfolio for mainstream clients. However it's not as bad as it might seem, most of the designers I know in the industry run sites or design plenty of other things than can be used in their portfolios, if they ever do decide to switch jobs. I've interviewed hundreds of people and rarely do I ever care about their previous companies, all I want to see is how well they can design, not who they designed for.

Granted I also wouldn't want to work for a company that would be biased against a designer who did adult work, so for me personally, it makes sense. If you have hesitations, I would fully recommend passing on working in adult.