We cannot represent "correct math" on a computer, because the set of reals is uncountable, but in 64 bits we can only represent finitely many numbers, not infinitely many, let alone uncountably many.

And given that we have to implement some sort of approximation to the usual math on a closed subset of the reals (rationals, even), why not extend it with some "numbers" (negative zero, plus/minus infinity, NaNs) that, yes, enable programming convenience?

There is a difference between 2.4987665678 as 2.5 and undefined as ±infinity especially because 1.0/+0.0 is +infinity and 1/0 is just an exception in Java. Shouldn't it be the same result?