Metadata is stored inside the image files themselves. You do lose Album sorting and the facial recognition database is trashed if you fish your files out of the Library, but basically all metadata is stored via EXIF and IPTC.