In my experience it's usually best to start with all the words. If you use a decent implementation that supports sparse vectors it's no problem, certainly not for these sorts of data sizes.

Usually you'll end up with a frequency threshold, but it's usually best to trim at the very low end --- like, words occurring 5 or fewer times. Further over-fitting can be controlled with regularization and parameter averaging.