when I download papers from arxiv I sometimes choose the LaTex version because it often comes with commented out ideas that didn't make it into the paper. The author thought process becomes clear. The Metadata helps me understand the whole thing quicker the same way the semantic web helps the machine.
Perhaps there is a clever philosophical analogy in there somewhere about "us becoming the machine" or "the map becoming the territory", but I can't put my finger on it.
I personally believe the key is in "information architecture". We have been conveying information as a linear sequence of words for so long that we don't know yet how to best exploit non-linear formats.
Programming languages harness the relation between specific instructions and the structure on which they are embedded; but this structure is oriented towards building a single executable block that controls a machine step by step. We have yet to build tools equivalent to IDEs to better exploit the overall structure of knowledge, but for the goal of understanding a topic at all levels of detail, bot at the local flow of ideas and the overall relation between its subtopics.
The first widespread step in that direction was the 1.0 static World Wide Web, and we have learnt a lot thanks to it so we can now improve upon it. I have great hopes in online notebooks and no-code spreadsheet-like tools as the basis of such information-processing environments.
I think we can use CTPH algorithms to fingerprint the data independently of whatever names are used, and then we can use that to find representations of the "same data" submitted by other users. Probably there would be some reputation stuff involved, a web of trust, etc. The flow would go something like this:
COGNIZE:
0. Encounter messy data in the wild (has pagination, timestamps of access, etc), need other representation (human/computer/whatever)
1. Calculate CTPH fingerprints, use them to search for link: miss
3. Clean data the hard way and publish canonical representation (ipfs?)
4. Generate missing representation the hard way, publish that too
5. Calculate the fingerprints common to both the missing representation and the canonical representation, and publish it as a "link" between the two. Unlike traditional web links, this one is bidirectional.
RECOGNIZE
1. Different user encounters "same" data in the wild
2. Calculate fingerprints, use them to search for canonical representation: hit
3. Find further links to see what other representations of the "same data" are available, download them if desired.
The fingerprint stuff works, but there's a lot of work left to be done re: mapping fuzzy hashes of "in the wild" data to cryptographic ones of "canonical representations" and finding ways to incentivize users to go through the hassle of the "cognize" step so that other users can benefit from the "recognize" step.
Sorry for talking your ear off, it just feels good to know I'm not the only one working on something like this, even if our approaches are quite different (mine works on PDF's only because it works on arbitrary bytes). Good luck with yours.
But I don't think it changes the main problem to be solved, which is that it's data consumers who want the semantic web, but so far it's been up to data providers to implement it. We need to be able to create links between data with without the participation of whoever hosts it.
But otherwise yes. I'll have to give that a closer look and use as much of it as I can.
Semantic web uses IRIs in triples. They serve as identifiers, not locators (URL). This is because semantic web talks about more than just files.
The Web Annotation spec talks about web resources which should be dereferencable. That is a naive assumption that makes the approach brittle. But one can annotate 'sourceDate' and 'cached' which indicate the date for the annotated version and a location for a copy.
When you're annotation a resource it's best to also store a copy. The Web Annotations specification allows this.
Do you know if there's any precedent re: people getting upset about that copy? If I annotate an article that was originally served with ads, or behind a paywall, I'll probably not include that stuff in my copy. If people then start linking to my copy instead of the original, I could imagine that feathers would be ruffled.
I figure if you configure the client to keep track of whose annotations you have a history of using, you've got a real granular view of which content providers to pay. I'm imagining a game where we all put $5 in at the start of the month and we all pay each other based on whose content we use the most. Some users will pay more than they make, others will make more than they pay.
This is exactly what we are trying to achieve at Anvil. 1. Provide the no-code tools to make it easy to convert existing PDF forms into web forms. 2. Share the web forms with perspective customers instead of PDF forms as email attachments 3. PDFs are generated as part of the workflow once the data is captured and represented in structured JSON. 4. (optional) request certification of the PDF via e-signatures
The end result is a JSON payload that can be shared via API as well as a static PDF that is stored for human consumption. In most cases, we find that our customers actually just use the PDF as an interface with legacy systems (IRS, Banks, Insurance Companies) that haven't yet figured out how to modernize to a data-first business model.
Of course this really only addresses PDFs that are used for information capture and transfer between two parties. But most PDFs that are not "standardized-forms" are made for consumption by humans not by machines (think ebooks, journal articles, graphics etc), and therefore having a JSON payload of the data attached doesn't really matter.
https://de.wikipedia.org/wiki/ZUGFeRD
My CodeDraw app also puts the source code in PDFs or PNGs that it generates.
Is this compatible with User Defined Language?
But: don't we need some way to prove that the data matches what is visually rendered in the PDF reader?
And if we can prove that the embedded data matches the rendered document, couldn't that same logic just be used in reverse to generate the structured data from the renderable PDF?
I do really like the idea of having a checksum of some sort if we end up embedding metadata like this.
But in the real world people are going to want to annotate the PDFs, there will be open-and-save cycles that add metadata and break the checksum, etc and so on, even without considering malicious actors. Restricting all that is maybe easy -- just reject anything that doesn't match the checksum, done -- but communicating that restriction to the users without making a mess of it is probably hard.
Long ago I wrote some PDF generating programs and it was a lot of fun, but the spec has evolved in ways I imagine would make it less fun today. Still, could be a cool thing, and I'd be surprised if someone hasn't already done a version of it somewhere.
[Edit: plus, whoever creates the one-way function is deciding what all the PDFs are going to look like, which means you will end up with many such functions to accommodate the different rendering goals, and then each validator needs to know each one and someone decides which ones to trust, and so on...]
Wait until you realize this applies to business presentations (Excel and PPT-authored-PDFs)
One day I will do it.
Speaking more broadly, whether we talk about HTML or PDF it's the same problem: documents should have two representations - human-friendly and machine-friendly until AI gets so good that only having the human-friendly representation is enough.