I don't understand how you handle raster images. You simply cannot invert them blindly. So it sounds like you just bite the bullet of never inverting raster images, and accepting that you false-positive some vector-based diagrams? I don't see how that can justify your conclusion "it wasn't necessary". It sounds necessary to me.
The choice to never invert raster images isn't a compromise, it's the design decision. The problem veil solves is exactly that: every dark mode reader today inverts everything, and the result on photos, histology, color charts, scans is unusable. Preserving all images is the conservative choice, and for my target (people reading scientific papers, medical reports, technical manuals) it's the right one.
It's absolutely true that there's a subset of raster images, like diagrams with white backgrounds and black lines, that would benefit from inversion. I could be wrong, but in my experience they're a minority, and the cost of accidentally inverting the wrong one (a medical photo, a color chart) is much higher than the benefit of inverting a black and white diagram, from my point of view. For now the per-page toggle covers those cases.
OK, so I did understand, but this sounds very hand wavy to me. You say it's a 'minority'; well sure, I never claimed that was >50% of images, so I suppose yes, that's technically true. And it is also true that a false positive on inverting is usually nastier than a false negative, which is why everyone defaults to dimming rather than inverting.
But you don't sound like you have evaluated it very seriously, and at least on my part, when I browse my dark-mode Gwern.net pages, I see lots of images and diagrams which benefit from inverting and where I'm glad we have InvertOrNot.com to rely on (and it's rarely wrong).
It may be nice to be able to advertise "No AI" at the top of the page, but I don't understand why you are so committed to biting this bullet and settling for leaving images badly handled when there is such a simple easy-to-use solution you can outsource to, and there's not a whole lot else a 'dark mode PDF' can do if 'handle images correctly' is now out of scope as acceptable collateral damage and 'meh, the user can just solve it every time they read every affected page by pushing a button'. (If Veil doesn't exist to save the user effort and bad-looking PDFs, why does it exist?)
Your approach with a classifier makes a lot more sense for the generic web, where you're dealing with arbitrary <img> tags with no structural metadata, and there you have no choice but to look at what's inside. PDFs are a more favorable problem.
A case where a classifier like yours would be an interesting complement is purely vector diagrams, drawn with PDF path operators, not raster images. Veil inverts those along with the text because from the format's perspective they're indistinguishable. In practice they're rare enough that the per-page toggle handles them, but it's the honest limitation of the approach.