After 500 years? Yes
Somewhere between is reasonable to debate, but many people won't even agree on the principal of self determination, at least not consistently. Personally I wouldn't put the threshold longer than a human lifetime
It gets all complicated and messy when war is involved, of course. I'm talking about peaceful transfers of ownership here.
My opinion is that what your distant ancestors did should not be held against you — go far enough back and pretty much everyone has some ancestors who were good people, and some who did flat-out evil things — and that what happened 200 years ago is not a good argument for overturning the status quo. Some exceptions could apply, but "my ancestors took this land by force X number of years ago" is true of just about everyone, everywhere. The only sane thing to do is to hold the past lightly and consider what is the right thing for the current inhabitants today.
But Africa is such a huge continent, with such a variety of countries (each with their own complicated history) that pretty much any generality you can make about Africa is going to be wrong about large portions of the continent, and about large portions of the history of its various countries. It's perhaps not such a good example to talk about an entire complex continent for a simple point, so I should stop rambling on about it. (But I do recommend finding a book about the empire of Aksum sometime. It's fascinating).
https://en.wikipedia.org/wiki/1967_Gibraltar_sovereignty_ref...
https://en.wikipedia.org/wiki/2002_Gibraltar_sovereignty_ref...
That's the same, whether it's Gibraltar, Scotland, Cornwall, the Canary Islands, Ceuta, Taiwan, Falklands, Cyprus, Texas, Point Roberts, Crimea, Canada, Greenland, etc etc.
We can argue about the thresholds needed, the length of time of residence ("squatters rights" etc), the minimum size of a given area, but the principal remains.