Why can't time begin at a point in the past, but extend infinitely into the future?
Well it might, but in what calculation would the fact that time extends infinitely be a factor? You can say all past time resulted in the current moment, but as far as I know there's no such thing flowing from the future into now. So any physics calculation you do will deal with finite amounts of time.
I won't pretend to know every possible calculation that might be desired, but I can imagine any sort of prediction or simulation about the future may want to use such an infinite view of time going forward.
> I guess my point is more that for a mathematician it would probably be obvious that when you talk about the limit of a divergent series it could be any imaginary or intermediary value.
I'm a mathematician and I might agree (not entirely sure what you mean). But -1/12 is a concrete value so it doesn't apply here.