From what I understand, it is very difficult to make large sections of pure graphene or long CNTs. As long as they're pure, they remain strong, and therein lies the difficulty of graphene and CNT production. Like you said, the impurities do weaken CNTs and graphene. The impurities can be contaminants which occupy a carbon location, or missing carbon atoms, forcing the lattice to reconstruct. According to [1], this can lower the tensile strength of a carbon nanotube by up to 85%. The problem with manufacturing long CNTs and large areas of graphene is that it is thermodynamically impossible to, on a large scale, have a perfect lattice. The imperfections will weaken the entire structure. So yes, as you go up in scale, the incredible/fantastic properties will just become impressive.
[1] http://en.wikipedia.org/wiki/Carbon_nanotube
*I currently study materials science and can produce literature upon request. Also this is from the top of my head so it may contain some errors.