But it's unlikely to increase without increasing the stats of these "magical players", as they need to have an impact of the game to increase the stats of other players and that counts as assists, rebounds caught etc.

Basketball is a small court, 5 people on each team, defense is very important but this hardly is football/rugby or even soccer level of defense everyone handles the ball quite a bit. Time with ball is also a stat that is counted, and often other stats are normalized to it.