The gas inside a standard party balloon is generally compressed 3% to inflate the balloon. This wipes out even the theoretical buoyancy of nitrogen. And trust me, there was never any practical buoyancy to begin with. You’d need a ridiculously large balloon in a room with impossibly still air and impossibly null thermal gradients to even measure the buoyancy of nitrogen vs air. The buoyancy of nitrogen vs air would never be perceptible to human senses in any real-world setting.

It would be the same as just filling the balloon with air.