It depends if they kept coming back. Smallpox worked to the settlers' advantage because colonizers were establishing themselves in the carribean and south america before any serious ventures into north america got going - so there was time for a pandemic to actually spread before folks really started getting serious about settling. I think with statistics and spread rates and all that it's likely that viking settlers would need to stick around for a decade or two to really see the effects in terms of population thinning.