Re:Ask HN: Have there been any innovations in crawling or da...
by PaulHoule on Friday June 02, 2023 @10:32AM
(#36165605)
In principle you could feed HTML into a transformer model and train it to do tasks such as classification (is this an "open access" scientific paper?) or extraction.
In practice the current models have a limited attention window so you can't feed a large HTML document into them. Chunking the document into smaller pieces is not attractive because you really want to have the start tag and the end tag of an element in the same window.
Long-range transformers are rapidly improving and somebody will soon "crack the code" and come out with something that will make the job easy even if it is slow and somewhat expensive.
Other approaches such as graph neural networks and systems based on classical machine learning are also in development so you will see some pretty awesome tools coming online.
In practice the current models have a limited attention window so you can't feed a large HTML document into them. Chunking the document into smaller pieces is not attractive because you really want to have the start tag and the end tag of an element in the same window.
Long-range transformers are rapidly improving and somebody will soon "crack the code" and come out with something that will make the job easy even if it is slow and somewhat expensive.
Other approaches such as graph neural networks and systems based on classical machine learning are also in development so you will see some pretty awesome tools coming online.