This is exactly what you should do. I've got quite a few friends in the swinger crowds, and if you're going to find a good, mature, financially responsible investor for a really good adult-oriented idea, your best bet is in the upscale swinger community.
I'll confirm that they're a mostly very open and very nice crowd, and there's a higher proportion of wealthy and intelligent people among them who would love to invest in a good adult-oriented startup - especially if it's a fresh idea.
I've also noticed that some of the consumers of the more specialised porn and upscale swingerclubs are wealthy individuals in their 40's and 50's. Naturally these people have a very open view on pornography and adult-oriented startups. If you try networking a bit in the upscale swingerclubs in the big cities and/or the well-run specialised sites you might find an investor. These people are mostly very open and very nice. Probably because they, like the gay community, are an outcast niche that has to constantly defend their sexuality.