Southern whites who weren't slaveholders tended to be economically dependent on slaveholders for their livelihoods, in a much more direct way than northern whites were dependent upon capitalists. The Southern economy was pre-capitalist in nature, dominated by a de facto hereditary, landed aristocracy, many of whom were descended from actual aristocracy who migrated to colonial America.

Also, there's a natural human reaction to seeing invading armies occupying your home town, regardless of the rationale behind that action or one's previous or preexisting attitude towards their present ruling class. I mean, there's a spectrum of reactions, but a large chunk of that spectrum includes hostility.