Available data points to it being more dangerous than the flu, especially to the elderly and immunocompromised. Your doctor might have been talking about the danger to you in particular, which is very low, similar to the flu, if you're young and healthy.