The flag link for comments is a little hard to find if you don't already know where it is. If you don't flag things, there's no way your flag privileges could be taken away.