Wild wild speculation: I'd imagine "rigidity" is a trade-off between weight and ability to withstand the given pressure differential for any two gases (outside & inside). A vacuum would undoubtedly give extra buoyancy but the additional weight required to achieve rigidity might not make the trade off worthwhile.