People who speak a romance language definitely have a stronger connection than many but Latin was far from the only language of the Romans. Greek was used extensively in the West and East alike, and there were regional languages like Gaulish, Aramaic and many many others elsewhere. Many of those who did speak Latin did so as a second language, with strong accents etc. So in many ways what is Roman is more about mindset, culture, institutions, aesthetic preferences etc than language. (and pretty much all of the Western world carry on Roman versions of all of the above, for better or for worse :P)