Evolutionary pressure doesn't factor into this. Imagine the virus doing a random walk over a bell curve of virulence. On average an outlier is going to mutate into a not outlier.

SARS-1 was an outlier, very deadly. SARS-2 s probably not an outlier.

That said I think a virus can evolve to the point where it's arguably a different virus in which case the median virulence can change.