News for hackers. Stuff that matters.
Re:SearXNG is a free internet metasearch engine by RaisingSpear on Saturday April 06, 2024 @12:05AM (#39949849)
Search engines: they scrape the web, but get narky when scraped themselves.
Re:SearXNG is a free internet metasearch engine by marginalia_nu on Saturday April 06, 2024 @04:33AM (#39950956)
Difference is a crawler paces the requests, respects robots.txt and rate limits, and doesn't typically invoke 50-100MB disk I/O per request.

Like I don't mind automated access to my search engine, I even offer a public API to the effect, that you can in fact hook into SearXNG. What I mind is when one jabroni with a botnet decides their search traffic is more important than everyone else's and grabs all the compute for himself via a sybil attack.