Lower genetic diversity, less species resiliency, lower growth for humanity across the board due to less humans exploring various problem spaces.... there are a lot of downsides to declining populations, depending on the surrounding circumstances/causes, just as there are upsides.