> I'm just not clear why this process is allowed (or more importantly, why it can't be removed).

Because we don't know how to make CPUs do pixel perfect images every single time. (I wrote a little more above)

Literally any kind of image or photo manipulation, from an MS Paint-like webapp to Instagram-like photo filters.

There are many legitimate uses. Vendors have experimented with making canvas readback opt-in (with a popup) but I don't know if it'll ever ship because it simply breaks too many websites. Sometimes it's used at page load to generate variants of a single image to reduce file sizes, or used by games to prepare image assets before they start up.

A lot actually, for example you can do easy image resizing (like taking a selected image and resizing before uploading to the server.)