I live in Toronto and have been to Ottawa many times, as well as tons of other Canadian and American cities. Downtown cores have some middle class folks in them, usually in somewhat luxurious condo buildings. But in no way does the interpretation of "where all the rich Canadians live" being downtown Ottawa make any sense.