Not all users can flag, and flagging is more to deal with spam and trolling. (The guidelines mention offtopic posts as well, but community consensus is that flagging is not downvoting.)

Flagging is not downvoting, but it is the sanctioned way to register your opinion that some content does not belong on the site. If your efforts to combat off-topic posts are limited to ranting in the comments after they're posted, it's too late.