This may be dependent on a country's tradition, but voting is not speech, it is one's most direct participation in democracy. Who doesn't vote ignores the system and thus (in my humble and personal opinion) is not a part of it.