But they were private property at the time they were stolen by the British. A government doesn't generally have the right to take objects from citizens simply because they are culturally important.

In modern terms, that would be like somebody from Britain stealing Warhol's "Eight Elvises". Then, the US government demands they be given to the Smithsonian instead of back to the rightful owner.