This isn’t surprising. An heritage is well, an heritage, which is coming from ancestors. Only very few people get to aquire culture heritage accross race boundaries (for example by adoption). It’s like saying the overlap of "people who celebrate Chinese new year" and "people who are not asian" is virtually non-existent to shame them; this make no sense.
Like, okay, sure, I guess the pointy white hood your grandfather wore when he was a Grand Wizard of the KKK is part of your heritage, but if you take it out and wear it around town, people are gonna be all "hey look at that racist jerkass", you know? And if I hung up a CSA flag outside my house in a big Southern city, all my neighbors would say the same thing no matter how much I tried to convince them it's just me celebrating my heritage.
Flying the Confederate flag is a sign to anyone else from the South that you are probably a racist. I would bet money that there is a strong correlation between “people who say it is a symbol of heritage” and “people who display the CSA flag”, because very few people are willing to come out and say “fuck yeah I am totally a racist and proud of it!”.
As a bonus, I would also bet that the overlap between “people who say it is a symbol of heritage and/or display it” and “people who are not white” is virtually nonexistant.