I did the same in Husavik some... 8 years ago in wintery conditions (april iirc, inland was completely inpassable and even ring road had heaps of snow) and they were fine, we saw plenty of them and pretty active (mostly mink whales). Maybe bad luck or they are quite seasonal?