Define the separation between your personal and professional life clearly. It's not a shitty thing to do.

I don't have any recommendations for which action to take next. But do try to speak to him honestly and open up to each other as deeply as you can not only to find out what action you should take next, but to come to a decision together.