I think LLMs will be capable of orchestrating between systems and maintaining a conscious narrative but knowledge graphs will solve the update and references problem.
If interested, a few rabbit holes to explore (no affiliations):
https://scite.ai -> best option for citation mapping, but same issues you described above
https://www.semanticscholar.org and AI2 -> the best group working on tooling in this space
https://www.weave.bio -> early startup trying to build this out
The hardest challenge in my view is solving the intermediate representation issue. You have to establish a DSL/nomenclature that provides the range required to represent a complete scholastic discourse while also being computable.
Right, you'd basically be writing an interpreter for English
Generally, what I found useful to build a graph between "topics" or entities was to use a HyDE[1] prompt to generate possible distinct definitions and then build a nearest-neighbor network from that. This successfully identifies related concepts in the truly abstract sense rather than literal entities.
I haven't uses Wikidata info yet, but hoping to expand to that in 3 or 4 months.
HyDE sounds like an interesting approach. All dense retrieval approaches suffer from the problem you outlined in the blogpost. Have you looked at keyword-based or late-interaction models for retrieval such as ColBERTv2[1]? I find that late-interaction methods seem to offer best trade-off between semantic intelligence (precision) and retriving relevant documents (recall).
This semantic extraction can be extrapolated to most any trainable context. A useful one I've worked with involved mapping supplier-partner relationships. A well built supply chain graph can identify every layer of risk in a single supply chain and provide the provenance to back it up.
IMO, the biggest blocker to more mainstream use of Knowledge Graphs (even in the commercial world) is an actually intuitive interface for knowledge exploration. The real market innovation behind GPT isn't its 175 billion parameters, its the prompt interface that makes ChatGPT so universally accessible.
Also, if you want to scale, LLMs are going to prove to expensive, so eventually data need to be logged somewhere to create a fine-tuned model that can do sub-tasks and ideally do them better. What do you think?
Depends on what you mean by "better". With more accuracy?
with all these strides in AI, GPT, etc, does anyone know of papers about extracting into graph or other form of knowledge representations deeper semantics ?
for example instead extracting what the paper is about, how it relates to other papers, nodes and edges linking to field specific terms/concepts mentioned, translation of paper specific terminology to commonly agreed terminology, why it's citing said references (is it an example of use a technique ? a proof for a claim ? something that's being refuted, etc ..), where in the text it's being used. and other kinds of "deeper meaning".
I've glanced at paper that seemed to do something remotely close to this but it was restricted to doing NER for only medical/biology terms.