I guess I focussed too much on the PDF aspect, my desire is some sort of system where if I annotate something, be it a PDF, website, image, text file or whatever, I'd like the contents of that annotation (with maybe some context/metadata around where that annotation was from) to go to some central place.
It's tricky though because a lot of apps are siloed, and the interaction ideally would be almost seamless. Like taking a screenshot.
Yep this is what I use evernote for. When I was working I had to consume and organize a lot of proprietary information and evernote was the only thing that worked. I use evernote clipper in my browser and forward from email. You can put additional information in the note. The generic "web aggregators" don't work for me because they don't work behind a fire wall.
Annotation itself is part of the PDF standard. It is defined well enough that all of the apps I use preserve annotation.
The parts that are missing are both provided by Evernote.
Evernote inserts the major annotations at the start of the file. You can then read a summary of annotations in any PDF app.
Evernote was also the first that automatically converted image based pdfs to "searchable pdfs". When this feature first came out it was awesome. Now I see it everywhere. My scanner automatically converts to searchable pdfs and I have an OCR converter that does the same.