* * *

NER, Entity linking, and relationship extraction definitely seem to be 'low hanging fruit' due to LLM improvements, but one of the big problems is that they really need a completely different architecture to limit the decoder vocab if using a decoder transformer for producing the set of sequences in relation extraction with specific entity ids. A Longformer with full attention to input sequence, and sliding window attention to a large dictionary could be a decent way to find tune a system like this, but there are few that try it. Unfortunately there's a lot of stupidity going around right now in thinking the answer is just to 'pRoOoMpT tHe LLm RiGhT', but that will always be exceedingly wasteful such that processing terabytes of files will be prohibitively expensive, and there's no guarantee the system will always restrict to the specific vocab and structure desired.

The images in radiology definitely make these types of things harder, and the sparsity is an enormous issue. However, working with some projects in this area, I don't think it's as impenetrable as a lot of radiologists in AI suggest. The main thing needed in the field is adoption of better techniques and architectures to deal with these problems.

I agree it's not impenetrable, that's why I'm working on this problem. What I disagree with is the "this is trivial" statements.

> A Longformer with full attention to input sequence, and sliding window attention to a large dictionary could be a decent way to find tune a system like this, but there are few that try it.

> Unfortunately there's a lot of stupidity going around right now in thinking the answer is just to 'pRoOoMpT tHe LLm RiGhT'

I agree with that this is not the right approach despite all the media hype (and that fine-tuning is), my research has been (more or less) attempting what you've proposed.

The problem right now is that there is no good dictionary (ontology). RadLex and UMLS have inadequate coverage. Working on a better one as we speak. Context length isn't an issue but Longformer/BigBird are good ideas, it's what I've tried as well.

I previously worked for a Radiology PACS and it's hard to get funding/interest to even tackle the problem. With how lucrative it could be, I would think that a corporation would be very interested in putting resources into it, but this has not been my experience.

No PACS that I know of even wants to tackle digital pathology in a significant way, which last I heard had about 5% adoption versus glass slides.

There are several corporations, some of which have raised > 100M, tackling this problem. AIdoc being one of the more notable ones.