Facebook could reasonably argue that by bypassing their established use policies for the like button, you are depriving them of value - in this case the value of the data that their JavaScript collects and sends back to them, and that you (the site owner) are being unjustly enriched through the use of their copyrighted image(s) on your site.

Except in cases of fair use, which isn’t nearly as broad as people think, the use of other parties’ images is subject to whatever licensing restrictions they choose to put on them. You can choose not to display their images if you do not accept those terms.

I feel like trying to sue someone for linking to Facebook would go down extremely poorly.

The reason sites want to have Like buttons is because they perceive value in that for the site. A C&D would be enough to bring anyone in to alignment that didn't have a wish for a very expensive day trip to court.

Especially when it's done this way for privacy considerations.