I guess I don't think where our ancestors came from is very important. I only know where my ancestors lived about 200 years ago and it is just general areas not any specifics. I don't know anything more than that. This is quite possibly about the same amount of years as many blacks whose ancestors came over during the slave trade.
I mean, the mere fact that you can trace your lineage back that far is indicative of the difference. Many people find their identity through their culture, and often times that perspective is gained by tracing their origins back generations. The foods you eat, the customs you share with your family, even your name.
Some people can trace their lineage back through dozens of generations. Other people can't see past a few levels up the family tree because their history was destroyed by a more recent slaver society. For some people, the traditions in their family are the traditions of their enslaved ancestors. The songs they sing were sung on the fields their ancestors were forced to work. The names in their family are the names forced upon them by their oppressors. Their family cook book contains recipes their enslaved ancestors used to make the scraps they were thrown palatable. Theirs is not an organic culture, but one that formed out of necessity due to the conditions forced on them by slavers (relatively) recently. So yes, I think it does matter that even if a person has never met a slave themselves, they can still feel the reverberations of slavery quite strongly.
More importantly, I can ask around in my family and find out family health history, how long people lived; I know where they came from and can find relatives. My African-American friends cannot all do the same. For some, their known family history only goes back to the last slave sale. They don't know where their ancestors came from in Africa. They have limited knowledge of family health history, compared to what I know. I know the language my great-great-great grandparents spoke; they don't. I can do research on historical foods from my area; they can't. With the advent of modern genetics, some can figure out some of that (look, maybe I'm Igbo, let me go to Wikipedia and look that up....) but it's quite different than being able to ask your aunt to set you up on a tour of where your ancestors lived in the 1700s and her being able to just look on her desk for those files.
I am going to go out on a limb and say 99% of people have had their ancestors enslaved. Is having an ancestor enslaved 150 years ago different than somebody who had their ancestors enslaved 200 or 300 years ago? If you, your parents, quite possibly even your grand parents never even met the slave in your family then it is irrelevant.