Re:SearXNG is a free internet metasearch engine
by RaisingSpear on Saturday April 06, 2024 @12:05AM
(#39949849)
Search engines: they scrape the web, but get narky when scraped themselves.
[ Parent
| on HN ]
Re:SearXNG is a free internet metasearch engine
by marginalia_nu on Saturday April 06, 2024 @04:33AM
(#39950956)
Difference is a crawler paces the requests, respects robots.txt and rate limits, and doesn't typically invoke 50-100MB disk I/O per request.
Like I don't mind automated access to my search engine, I even offer a public API to the effect, that you can in fact hook into SearXNG. What I mind is when one jabroni with a botnet decides their search traffic is more important than everyone else's and grabs all the compute for himself via a sybil attack.