As long as your aren't the current group it is popular to pick on and you don't anger (or make fall in love) anyone with significant power, you'll probably be fine. I'm sure there were plenty who didn't care about slavery as long as they weren't the slaves.