You're totally right, and this exactly the sort of thing Wittgenstein set out to show. The trouble is that people have an idea of what a "heap" is and what "evil" is and what "knowledge" is and a lot of philosophy is concerned with pinning down those notions or at least saying something concrete about them.