I would argue most people care about GDP growth targets but they just don't understand them that way. GDP growth is what leads to improvements in real human quality of life. It is your ability to get healthcare, housing, food and all the other things that make life a lot less painful.

I would rather live in a society with a higher GDP than less (all things considered) and so would most people (as evidenced by population shifts across the world). If the GDP does not grow to at least match the population changes, the society would be poorer and have worse outcomes overall.