I don’t see the controversy of your example because since there was no UNESCO and other global agreements in the 1500s, the indigenous people of this area are still around and so it would be the local Native American tribes that can have a legitimate claim on their heritage.

Lots of native american tribes committed genocide and destroyed other tribes. If Tribe A made some artifacts, and then was genocided out of existence by Tribe B, which was then conquered by the Tribe C (USA). Would you want Tribe B to have ownership of the artifacts of tribe B?

Including your 'tribe C' (the USA) with two indigenous tribes (A & B) in your example is highly disingenuous. And self-serving to your contrived hypothetical.