I still think that the changing temperature was likely a factor but perhaps just a small one. It certainly would have got harder to eek out an existence on the island.

As you say though a bigger factor was Portuguese traders gaining access to African elephant ivory which was easier and cheaper to procure than walrus ivory which was the cash crop of the Greenland settlement.