News for hackers. Stuff that matters.

A comment on A search engine that favors text-heavy sites and punishes modern web design. Parent comment

Re:A search engine that favors text-heavy sites and punishes... by marginalia_nu on Thursday September 16, 2021 @10:05AM (#28551953)
It's pretty much all bespoke.

I use external libraries for parsing HTML (JSoup) and robots.txt; but that's about it.

[ Parent | on HN ]
Re:A search engine that favors text-heavy sites and punishes... by soheil on Thursday September 16, 2021 @11:26PM (#28560865)
What was the starting site you fed to the crawler to follow the links from to build the index?
[ Parent | on HN ]
Re:A search engine that favors text-heavy sites and punishes... by marginalia_nu on Friday September 17, 2021 @07:19AM (#28564020)
Just my (swedish) personal website. The first iteration of the search engine was probably mainly seeded by these links:

https://www.marginalia.nu/00-l%C3%A4nkar/

But I've since expanded my websites, so now I think these play a decent role in later iterations, although they are virtually all of them pages I've found eating my own dogfood:

https://memex.marginalia.nu/links/fragments-old-web.gmi

https://memex.marginalia.nu/links/bookmarks.gmi

[ Parent | on HN ]