The fighter can only slow down to like 180 knots, while the balloon might be blowing around at 10-90 knots depending on where exactly it is. How close do you have to be with the Mark 1 eyeball to actually disambiguate this kind of target, how long would you actually be at that range in a fly-by, and how many fly-bys are you willing to do before "whatever, we will just apologize to the NOAA or a college somewhere" and shoot it down.

Maybe the targeting pod could give you a zoomed up view, but it's not really designed for nearly stationary, high altitude objects.