The same objection would theoretically apply to voice, where the English passive voice must be periphrastic too, but in that case everyone agrees that this is a distinction of voice and the difference between inflection (where grammatical meaning is expressed by changing the form of a single word) and periphrasis (where grammatical meaning is expressed by combining multiple words) isn't relevant. This is just an inconsistency in modern theory, which probably arose because voice isn't relevant to semantics at all.
Ignoring the modal auxiliaries, English would still have moods, subjunctive ("We demand that Robert be ejected from the book club") and irrealis ("If Robert were to be ejected from the book club, ..."), but neither of those is in a particularly robust state in the modern language.