So why not have a certain epsilon, below which you can turn the connection off altogether? (meaning the back-propagation would only apply to the remaining connections) To avoid getting stuck in local minima you could occasionally re-initialise them with a random small value.

Again, zero background in machine learning here. It's a sincerely naive question to which I fully expect a "we've tried that with, methods X, Y and Z are most famous and this is how they work out in practice".