News for hackers. Stuff that matters.

A comment on A search engine that favors text-heavy sites and punishes modern web design. Parent comment

Re:A search engine that favors text-heavy sites and punishes... by marginalia_nu on Sunday September 19, 2021 @05:17PM (#28588007)
About 21 million, and I'm crawling myself.

Classification is really hard. I'm struggling with it myself, as a lot of like privacy policies and change logs turns out to share the shape of a page of text.

I'm thinking of experimenting with ML classifiers, as I do have reasonably good ways of extracting custom datasets. Finding change logs and privacy policies is easy, excluding them is hard.

[ Parent | on HN ]
Re:A search engine that favors text-heavy sites and punishes... by ad404b8a372f2b9 on Tuesday September 21, 2021 @06:10AM (#28603540)
If you're open to sharing your index I could make a classifier for you, I do this for a living. It's more of the indexing and search engine part which have been a problem for me. That's why I'm working from commoncrawls.
[ Parent | on HN ]