That is the real problem.
The problem there being the in his case "enough people" would have been a great many people and contrary to what we tell ourselves us humans don't tend to work so well in large groups especially when you are relying on the rest of the group to stand firm.
That is the point of the quote: many people didn't take action because of the risk to themselves if not enough other people took action too. If everyone speaks out you have some safety in numbers assurance, even if a small proportion of people meet with reprisals. If only a few speak out with you, you are guaranteed to be picked off in short order.
That is not a risk we would naturally consider, and as the quote suggests this locally greedy algorithm for deciding when to act and when to keep shtum falls easily into the hands of a growing oppressive force.
Then they came for the trade unionists, and I didn't speak out because I wasn't a trade unionist.
Then they came for the Jews, and I didn't speak out because I wasn't a Jew.
Then they came for me and there was no one left to speak out for me.
Martin Niemöller (1892–1984)