But in those 324 years a lot happened that make the case of a direct lineage questionable. The Gothic wars destroyed a lot of remaining infrastructure and Roman culture, the Roman senate was no more after 600 CE, the long term population loss, and the Christian Pope was the defacto ruler of the city (if you could still call that what was left a city).
The coronation of Charlemagne was also considered illegitimate by the Eastern Roman emperor at first, until they couldn't ignore Charlemagne's military power anymore.
It's not any more or any less the end of the Western Roman Empire than the political troubles that occurred in the earlier and later centuries. The Gothic, Frankish or Vandalic kingdoms very much thought of themselves as belonging to a Roman world.