This isn’t remotely true. The Berber-speaking population of North Africa, i.e. the autochtonous population that was living there before the Arab invasion (and, of course, before Roman influence) is generally swarthier than those who can demonstrate pure Arab descent.