A search engine that favors text-heavy sites and punishes modern web design

Re:A search engine that favors text-heavy sites and punishes... by ad404b8a372f2b9 on Thursday September 16, 2021 @09:31AM (#28551534)

Very cool project! How many websites do you have in your index? And how did you go about building it?

I've been working on an engine for personal websites, currently trying to build a classifier to extract them from commoncrawl, if you have any general tips on that kind of project they'd be very welcome.

[ Parent | on HN ]

Re:A search engine that favors text-heavy sites and punishes... by marginalia_nu on Sunday September 19, 2021 @05:17PM (#28588007)

About 21 million, and I'm crawling myself.

Classification is really hard. I'm struggling with it myself, as a lot of like privacy policies and change logs turns out to share the shape of a page of text.

I'm thinking of experimenting with ML classifiers, as I do have reasonably good ways of extracting custom datasets. Finding change logs and privacy policies is easy, excluding them is hard.

[ Parent | on HN ]

Re:A search engine that favors text-heavy sites and punishes... by ad404b8a372f2b9 on Tuesday September 21, 2021 @06:10AM (#28603540)

If you're open to sharing your index I could make a classifier for you, I do this for a living. It's more of the indexing and search engine part which have been a problem for me. That's why I'm working from commoncrawls.

[ Parent | on HN ]