If I were your landlord and you refused, I would sue for violating a no-pets clause in the lease to get a court order to allow the inspection. Or perhaps just initiate eviction proceedings for the lease violation.

It is what it is.