I agree, my objection to the serious (say, Bostrom's) version of the argument is exactly along that line. When we add new entities to our set of those worthy of moral consideration, we don't imply they should all be given equal value. In less abstract terms, it's perfectly consistent to hold the following two moral positions: "animals should not be mistreated gratuitously", and "it is morally permissible to eat meat"; this would amount to having an utility function where animals have some nonzero weight, but not enough weight to offset reasonable human desires.
Likewise, that averting existential risks should trump all other considerations would not only require that future humans have some nonzero moral value, but that their moral value ought to be high enough that everything else pales in comparison.
Something like "my concern with future humans decreases exponentially with time", like you propose, still leaves us with a "normal" position: "sure, we should care somewhat about the future of mankind, but not, like, above all else". Both the premise and the conclusion of this "boring" position seem much more reasonable to me.
I do find it interesting that most people don't object to the idea of thinking about what is best for your hypothetical grandchildren.