Flagging is not downvoting, but it is the sanctioned way to register your opinion that some content does not belong on the site. If your efforts to combat off-topic posts are limited to ranting in the comments after they're posted, it's too late.