Of course those are just anecdotes, but the data confirms it. Crime rates have declined meaningfully over the years. Gentrification made these nice places to live, but the decline in crime is why they are less scary. Unless you are trying to imply that crime isn't scary?
I've known people that have lived in Parkdale for thirty years, and while it's better now, less petty crime, it's never been an especially violent place. "Sketchy" in Toronto means places where you can't leave your bike unattended for six seconds or where your garage will be ransacked by people every couple of years if the door isn't locked properly.
Gentrification pushed the troublemakers further away, jacked up rents, and changed the demographics substantially. It's a whole different town in some places.
There's never really been "scary" places to live in Toronto apart from a few neighbourhoods and housing projects. The places that people describe as scary just have a lot more "character" than other parts. You'll see mentally ill people on the street, those with physical disabilities, those who are dirt poor. In other parts of the city they're swept away or ignored.
What you're describing is gentrification. Gentrification has happened.