It's impossible to act in a world of limited resources without "harming" other people. The division between speech and action is practical under the circumstances.