News for hackers. Stuff that matters.
Re:SearXNG is a free internet metasearch engine by marginalia_nu on Friday April 05, 2024 @07:17PM (#39948416)
Virtually all public search engine endpoints see an insane amount of bot activity, often several queries per second.

If you delegate queries to e.g. google or bing at that rate, you'll be ip blocked in a heartbeat.

Re:SearXNG is a free internet metasearch engine by mostlysimilar on Friday April 05, 2024 @07:21PM (#39948447)
Ah duh, for some reason my mind didn't go to hosting the search instance locally and I misunderstood.

btw thank you for Marginalia! The spirit of the small web is very important to me.

Re:SearXNG is a free internet metasearch engine by RaisingSpear on Saturday April 06, 2024 @12:05AM (#39949849)
Search engines: they scrape the web, but get narky when scraped themselves.
Re:SearXNG is a free internet metasearch engine by marginalia_nu on Saturday April 06, 2024 @04:33AM (#39950956)
Difference is a crawler paces the requests, respects robots.txt and rate limits, and doesn't typically invoke 50-100MB disk I/O per request.

Like I don't mind automated access to my search engine, I even offer a public API to the effect, that you can in fact hook into SearXNG. What I mind is when one jabroni with a botnet decides their search traffic is more important than everyone else's and grabs all the compute for himself via a sybil attack.