That was my thought as well, it would be nice if someone more knowledgeable came and explained us why it's stupid/doesn't work

Weight pruning is already a thing. See, for example: https://www.tensorflow.org/model_optimization/guide/pruning

Besides pruning (which is what you describe) you might find the "lottery ticket hypothesis" interesting. The idea is essentially that in a randomly initialized large network there is a subset of weights that are already closeish for a given task and training helps tune that and suppress everything else, and that knowing this you can actually get better results by pruning before training.