Dying raises the average age if you die below the average age, and lowers the average age if you die above the average age. Thus, a 16-year-old dying can only lower the average age if the average age was already less than 16. A four-month-old dying can only lower the average age if the average age was already less than 4 months. (Which, obviously, it can't be.)
Dying lowers life expectancy regardless of when you do it, and it has a stronger lowering effect the earlier you do it. The dying four-month-old has a really strong negative effect on life expectancy, but a strong "positive" effect on the average age. They're orthogonal[1] concerns.
[1] OK, not technically orthogonal. One will help predict the other, especially if you know the average age. But the point remains that a negative effect on life expectancy can be associated with any effect at all on the average age, because all effects of death on life expectancy are negative.
Maybe it should have been a surprise to you, because dying young is not the reason Africa is the youngest continent. They have young people because Africans are still having children, unlike much of the rest of the world.
Young people dying tilts the population balance older, not younger.