Language families refer to the provenance of a language, the oldest vocabulary. That doesn't automatically mean it will be easy or hard to learn if you speak another language within or without that family. For instance if you speak Spanish you may find it easier to learn Tagalog than if you speak Malagasy (Madagascar), even though Malagasy and Tagalog are in the same family and Spanish is not.