Square-cube law. Mass scales with surface area, lift scales with volume. Large balloons (conventional, not vacuum) are simpler and more efficient than small ones.
As to materials, a tremendous problem with areogels are that they're exceedingly friable. Any friction, flexing, or stress will crumble the gell to a powder. I suspect this is why the gel hasn't taken off as it had been projected to. I did some very-early 1990s work in the space and aerogels were a noted emerging technology thought to have applications in, e.g., mobile home, pre-fab housing, and RV designs. For the most part, fibre-based insulation or expanded-foam (polystyrene) insulation remains the standard, largely because road and other vibration don't reduce your insulation to a few inches of fine dust at the bottom of wall cavities.
Material properties are complex, and represent interesting trade-offs between afforded capabilities and imposed constraints.
The idea sounds relatively sound to me. One could construct a sphere out of aerogel cones, and the external pressure would reinforce it. Not sure how practical it would be to build such a thing, though.
At some point, since volume grows slower than volume, I expect you can make the envelope relatively thick without sacrificing density. That's also the idea behind Cloud nine: https://en.wikipedia.org/wiki/Cloud_Nine_%28tensegrity_spher...