I love it here too, but as someone young(ish) with a stable job and who got into a home outside the downtown core before the latest boom.. it's a lot easier to look past Victoria's faults.
No city is perfect I suppose.
The usual Canadian caveat definitely applies: tech salaries are way lower than in the US. And the discrepancy between those salaries and cost of living here is pretty extreme even by Canadian standards.
We're also suffering from the typical West Coast city problems of homelessness/drug abuse/mental illness/property crime.
But, on the flip side: you get the best weather in the country, your backyard is some of the most beautiful nature on the planet, we have a vibrant and unique and walkable downtown, we have tons of great biking infrastructure, and there's reasonably convenient access to both Vancouver and Seattle when you need them. I love it here; it's a city with a lot of problems but a lot of potential.