Adding more ground stations and building more satellites does reduce latency, because there will be a satellite closer to "halfway between" you and a ground station, which defines the shortest possible path.

If there is just one (non-stationary) satellite in your sky, the geometry is probably pretty crummy, so light-speed delay is greater. Also, link budget is worse, so probably data rate is lower too.

Yes, once you have a ton of satellites, adding more doesn't improve latency much.