Much like surface trash cleanup, the perfect is the enemy of the good. The goal is "less junk on the streets", not necessarily "absolutely squeaky-clean streets". First get rid of that rusty fridge that someone dumped in the woods before we deal with all the cigarette butts.

Let's focus on the more problematic, larger objects first, that are at risk of creating much more debris if hit.

What's the minimum size at which debris is dangerous?

Since everything is moving at orbital speeds, and generally at an angle to each other, i understood even small fragments move at a few km/sec relative to what they hit, so don't need to be particularly large.

True, but I imagine that, the smaller the object, the smaller the chance that it will be hit/it will hit something, and a smaller amount of fragments that the hit can generate.