Re:SearXNG is a free internet metasearch engine
by mostlysimilar on Friday April 05, 2024 @07:21PM
(#39948447)
Ah duh, for some reason my mind didn't go to hosting the search instance locally and I misunderstood.
btw thank you for Marginalia! The spirit of the small web is very important to me.
Re:SearXNG is a free internet metasearch engine
by RaisingSpear on Saturday April 06, 2024 @12:05AM
(#39949849)
Search engines: they scrape the web, but get narky when scraped themselves.
Re:SearXNG is a free internet metasearch engine
by marginalia_nu on Saturday April 06, 2024 @04:33AM
(#39950956)
Difference is a crawler paces the requests, respects robots.txt and rate limits, and doesn't typically invoke 50-100MB disk I/O per request.
Like I don't mind automated access to my search engine, I even offer a public API to the effect, that you can in fact hook into SearXNG. What I mind is when one jabroni with a botnet decides their search traffic is more important than everyone else's and grabs all the compute for himself via a sybil attack.
If you delegate queries to e.g. google or bing at that rate, you'll be ip blocked in a heartbeat.